Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I am working on a small project where I have to submit a form to a website.
The website is, however, using onclick event to submit the form (using javascript).
How can the onclick event be simulated in python?
Which modules can be used? I have heard about selenium and mechanize modules. But, which module can be used or in case of both, which one is better?
I am new to web scraping and automation.So,it would be very helpful.
Thanks in advance.
Ideally you don't even need to clicks buttons in these kind of cases.
All you need is to see at what webservice does the form sends request when clicked on submit button.
For that open your developer's control in the browser, Go to the Network tab and select 'preserve log'. Now submit the form manually and look for the first xhr GET/POST request sent. It would be POST request 90% of times.
Now when you select that request in the request parameters it would show the values that you entered while submitting the form. Bingo!!
Now all you need to do is mimic this request with relevant request headers and parameters in your python code using requests. And Wooshh!!
Hope it helps..
There is no silver bullet in simulating onclick events on a web page. It is pretty much use-case specific, but here are some points and guidelines.
In general, there are two approaches:
use browser developer tools, open the network tab, make the click and see what request is being sent to the server. Then, simulate this request in Python, with, for example, requests.
use selenium which would fire up a real browser where you would find the specific element and click via .click() method
mechanize would not execute/trigger the onclick function, because executing onclick requires executing javascript which mechanize cannot do.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 1 year ago.
Improve this question
I explain my problem to you:
When I do PHP curls on some site or want to display the source code of the page element is missing a lot. I think some part is called by a script or something. Could someone help me view the entire code with Curl PHP.
To duplicate my problem go to Facebook or LinkedIn and right click on the page and "View the source code of the page", in this you don't see all the page content but when for example you right click and "inspect an element" You can.
Thank you in advance
CURL can't do this. It's not designed to render HTML or execute JavaScript.
A lot of the content on Facebook, LinkedIn, Twitter and many other pages is loaded through different ways. (like fetch()-requests or WebSocket-Events)
Some nodes you can see in the inspector are not part of the original document (which you are viewing with "view source" or curl downloads). What you see on the inspector is everything currently held in memory, which was partially (or completely) created with a scripting language.
This is basically done to
reduce the load on servers as it doesn't have to generate the whole page on every request
reduce traffic on clients and servers (no need to reload the header-data and/or scripts over and over again)
If you need data from a rendered site, you should either check if the website provides an API which gives you the data you are looking for or use one of the cli-rendering-engines from this answer.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
Tried to get links to all issues of a publication for a given year, from this link: http://www.vetsci.org/journal/list.html?pn=vol&year=2018.
E.g: "Jan. Vol. 19 No.1. 1~160", which has an 'onclick' event.
edited
I would appreciate any help with how to do the following:
first, activate the onclick event;
second, get the link of the page to which the onclick event is leading the browser -> I have used the developer tools after clicking, to see the link of the new page. However, since this is no different to manually clicking and copy/pasting the url, I would like to know if there is a way to automatically read these links.
edited
Hope I am not spamming, since there are a couple of other threads on issues with getting popups and similar, but those seem like somewhat advanced stuff - at least from where I am standing :)
If you want to activate the event itself in any situation, you'll need something like Selenium. If you want to retrieve the page in the example you gave, inspection of the code shows that the onClick event loads the list.html page with parameters like so:
http://www.vetsci.org/journal/list.html?s_v={volume}&s_n={number}&sort=start_Page*1&TG=&pn=vol&year=2018&sm={4th argument}
I can't quite tell what the 4th argument is supposed to be because it's blank in all the examples you gave. If you do a GET request on that URL, you can use something like BeautifulSoup to parse the page.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
I have a requirement where, Users inputs something and submit then angular does a service call which returns if users input is valid/invalid. If valid take user to success page, if service returns invalid/bad input then user needs to be taken to failure page and should never be able to go back(using back button) or user refresh the page, user should still be on same page. Only option provided to the user should be close browser, there by not allowing using to submit one more request(leading to service call).
You cannot prevent the user from not running or editing client-side JavaScript code. Since this is a security requirement against the interest of that particular user, the solution must be server-side:
On the server, after getting the wrong answer, mark the user's profile as such. You may need an additional table in your database joining users and questions for that.
Whenever the user loads the question or submits it, first check the user's flag. If it's set, error out immediately.
Note that this behavior is quite hostile to the user. For instance, the user may accidentally touch Enter too soon, and will be shut out by your system.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I am very familiar with programming, MySQL, PHP, JavaScript, jQuery, HTML and CSS, and this issue of mine has nothing to do with what I just mentioned. It all boils down to three things:
Defining a menu layout with link options.
Rendering a dropdown menu on-demand with position calculated live.
Determining if the current user with current session is owner of the object clicked.
Nr. 2 is not a big problem, very simple indeed. Nr.1 & 3 are a bit problematic. And it is this part that I do not fully understand, I know that for Nr. 3 Ajax can be used to communicate with the PHP in the backend and determine ownership of an object through IDs.
As for Nr.1 should this be put in PHP or as a JavaScript AMD definition, I am not so sure about this myself.
I hope things are much clearer now.
Yes it is. Ajax can be used to load any data (html also) it depends on the programmer that what he wants to do with the data. For example, if you send a request to server to send some html, and in response it sends HTML to you then you can just simply render that html code in your DOM. It is as simple as that...
It actually depends on the requirement that you should use an ajax call or just use jquery to append or show some html.
(For simplicity and for your question):
Use Ajax:
To get data from server.
Append html using ajax if the html contains the data which needs to be get from the server.
Use Jquery:
To append/show html which is present on page ,no need of ajax call.
Like for example in above quora dropdown ,
-There is no ajax when you click on that "..." image(Although you can use ajax here but it depends on the application requirement,here quora is getting this drpdown values on page load).
-There is ajax when you click on "Make anonymuous" ,"Add to reading list" etc etc.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking for code must demonstrate a minimal understanding of the problem being solved. Include attempted solutions, why they didn't work, and the expected results. See also: Stack Overflow question checklist
Closed 9 years ago.
Improve this question
Hi and thanks for your time. I have an landing page selling a product (Wordpress Course, for example). When someone clicks on the buy button, he´s redirected to a payment service (paypal like).
What i need from you is a simple thing: an idea (or even a clue) on how to count the number of times the submit button (buy button) is clicked.
I can easily imagine how to count page views with PHP, but is it possible to redirect him to the third party and at the same time reload the initial landing page, to execute my SQL Update Query to count one click, or something like this?
You could use Ajax.
When the button is clicked send an AJAX request to a php page just like others have suggested. However since it is asynchronous, your users won't even notice it.
In terms of code you could just send the ajax request, then redirect to the payment page. Easy and done.
EDIT---
To get info on ajax:
http://api.jquery.com/jQuery.ajax/
http://www.w3schools.com/ajax/
Have the link go to a page on your end, i.e. click.php, that does the SQL update, then redirects to the actual URL you wanted them to wind up on.
Or, just create a bit.ly (or any other URL shortener) URL and use their built-in click tracking for free.