I am trying to parse a whole website
Target Link: http://targetsite.com
which has lots of child links and I want to parse all its child links.
Links within the Target Page:-
http://targetsite.com/childpage/1
http://targetsite.com/childpage/2
http://targetsite.com/childpage/3
The problem I am having is that if I am directly opening these links for example -
http://targetsite.com/childpage/1
using either Curl or Python urllib2 or Python Selenium.
Then the website is redirecting me to CAPTCHA Page first to Prevent Automated Malicious Programs from parsing that page.
But if I open that link by click it in then its Opens without redirecting to CAPTCHA page.
Opening http://targetsite.com/childpage/1
By clicking this link from http://targetsite.com main page.
And then it displays that page.
I want to know how the website is tracking whether a link has been
opened directly or by click action.
Suggest a suitable way to do this kind of Web Scraping automatically in Python
NOTE:-
I have also tried it by just copy pasting that link to browser and opening it directly without using click action in that case too its redirecting to CAPTCHA Page.
Related
I can't redirect user on new page with javascript variants code:
window.location.href=url
window.location.replace (url)
window.location.assign (url)
The problem of redirect only on android, on other platforms there is no problem with going to another page.
How to fix it on android ?
View problem:
when going to any page
In order to solve the problem you have to understand it. This happens on the Android version of Telegram in WebApps. When you click on a link, it does not open in the WebApp area but in the browser.
This browser is Telegram's internal or external browser, depending on what was indicated in Setting/Chat Setting/In-App Browser.
This bug occurs regardless of the link. So not just with your Javascript calls. A simple produces the same effect.
It seems that it comes from a bug at the level of the WebView therefore in the code of Telegram. So a change in the code of your WebApp (another Javascript function for example) will not change anything.
Solution
The solution that I tested and that works is to use an iframe. Instead of having for example index.html which has links to page1.html and page2.html, you rename your index.html to menu.html and you make an index.html page which has an iframe containing menu.html.
If you put a borderless and full-surface iframe, the user won't notice anything.
And no need to go from menu to page1 or page2 with an iframe.src: direct Javascript links between pages that are in the iframe work perfectly, just like simple href links.
Just one point: you have to perform a window.Telegram.WebApp.ready(); in your first page so the one with the iframe (and also in the others)
I am developing a website and having trouble figuring out what I can use to accomplish this requirement.
I need a website where links in the document load the contents of page that was clicked.
I am thinking about using angular.js but how might a user get back to the back by entering it into the url.
Example of what I am looking for:
You are on www.example.com
You click the link to www.example.com/profile/1234.
The page doesn't reload but loads the contents of the new page.
The static element at the bottom of the page doesn't change the the rest of the page does.
The url has also changed and you have the history of being at www.example.com
You can also load the exact same page by pasting the url www.example.com/profile/1234, it also has the same bar at the bottom.
You could also say I need something similar to youtubes website. You click a link and it loads only some of the page. But if you re-enter the url you get all of the page.
Thanks.
I have a hyperlink in my web page that opens a Word document however before the document is open I need to perform some other tasks. I substituted the hyperlink for a button and this successfully runs my code but how can I then open the Word document in a browser agnostic fashion, i.e. similar to the anchor element;
Produce Report
I have tried several searches for this problem but most of the responses suggest using something like "window.location" top open the file but I don't want to load the file in the browser.
Any suggestions would be greatly appreciated.
I couldn't find anything about this topic, maybe because I'm not a good english speaker and can't find the right words to search for in google therefore.
I'm currently working on an Chrome browser extension which enables me to search for user accounts on a specified web page by using the context menu. Example: If I select the text name123 on any webpage and click the context menu entry, a new tab http://www.webpage.de/user/name123/ is opened.
Now I want to extend the extensions capabilities: I want to search for email addresses, too. For that I can use http://www.webpage.de/search/name123#mail.com/ which brings up a list of all user with the email address name123#mail.com. From there, I could extract the link to the user account as follows:
document.getElementsByClassName("xyz")[0].href
My question: Can I skip the loading of http://www.webpage.de/search/name123#mail.com/ and the "extraction" of the url to the user account? Respectively, can I hide the procedure of opening this additional web page? Does JavaScript support any kind of "preloading" of a webpages content, without being displayed in the browser?
If you can parse the required link from raw HTML, you can fetch the page via an XMLHttpRequest in the background page, examine the result and then open the real profile page.
If the page is dynamic and you need its scripts to run before you can extract the link, you can load it in an iframe in the background page and examine it.
You can do it.
Add permessions to your manifest.json file: 'http://www.webpage.de/*'
Create neccesary ajax requests from background script.
I am trying to add a Pinterest button onto a product page for a single image in a Facebook Tab app.
The Pinterest code is grabbed from their goodies section, and must include a URL, the Media URL and the Description.
I am not that proficient with Javascript. I am trying to pull the URL and insert into the Javascript button for a single item. Please see below; Thanks.
Pin It
Since you trying to get Pinterest button working in Facebook Page tab this will not work due to inability to know which Page you're on in JavaScript.
This info is passed to server with signed_request. You have couple of options to do so:
Build the link to page that will be linked to "Pin" server-side
Pass information about Page and Page Tab to client side and build the link here.
Beware that this is just a link and it will not looks like Pin-It button until you including Pinterest JavaScript (//assets.pinterest.com/js/pinit.js) on a page. If you build that link in Client Side you must include that JavaScript after, so link will be replaced by iframe with actual Pin-It button.