I'm trying to do web scraping using puppeteer. The element I need to handle loads lately. When I click on the search button the result loads in AJAX and I need to pick the element I am trying to pick is in the search results but not in the initial load of the page. The page screenshot it is producing contains search results too and if it output the HTML source I can see the element there too. but not sure why I cannot pick it.
You can use await page.waitForSelector(cssSelector); to ask Puppeteer to wait for any element to be displayed in the UI before continuing on to further steps in your script. By default, the timeout for the wait is 30 seconds but you can set it to any timeout you wish.
So in your case I would:
Enter your search text into the search bar.
Click on the search button (this will execute your AJAX call to load the results).
Use await page.waitForSelector(cssSelector); to ask Puppeteer to wait until some element you are sure will be displayed in the UI after executing the search is visible.
Now that Puppeteer has registered the element as visible, you know that any actions you wish to perform on it will also execute correctly.
What you might find happens, if you don't use that waitForSelector() call is that the element is displayed but Puppeteer will timeout, for example, if you wish to execute a click command on an element. This is because the timeouts for click events (and other Puppeteer events which interact with elements) is very short and sometimes the script (especially in headless mode) can move to the next instruction too quickly to allow for the UI to update fast enough to keep up.
So by adding the additional waitForSelector calls, you're also making your scripts much more robust. Especially when data is being generated dynamically as they are in your case.
Related
We've been browsing PDFJS's documentation looking for event callbacks for when the document has finished loading all of its text and is ready for searching. We have a search that searches a PDFJS document for any given text, when you try to perform a search as soon as the page loads, when there are plenty of documents to be loaded, we get different search results and it looks like its because PDFJS is still loading the background text information we're searching through. We found that there's an event callback for onPageRendered, which we use so users can't search before the pagerendered callback is hit, and that's enough to make the user wait til the rendering of the PDF is finished to display the user search panel. However, it seems there's still background text loading happening after the pagerendered event completes, and we can't find another event to bind a callback for when its text content has finished loading. Does anyone know how we can handle this issue?
This is the event we're currently using:
eventBus.on(
'pagerendered',
_.debounce(() => {
}, 1000, {trailing: true})
)
I am a lowly operations employee without authorization to change the programs and permissions on my machine, and I would like to automate some highly repetitive data entry. I know there are a lot of programs that can do that, however, for the sake of this discussion we'll assume that I'm not allowed to have any of them and I can only script through the debug F12 menu in Chrome. I also probably don't understand half of these words as well as I should.
I have to run test cases on a third-party vendor's highly dynamic website, and I've already successfully written javascript which adds texts to elements in the DOM and presses the "next" button.
The problem is, upon .click()ing the "next" button, it takes time for the page to update, and the update creates new elements which weren't in the DOM when the script was initialized. I need to find a way to delay the execution of the script until the DOM contains all the elements I need to update.
As a really, really crude proof of concept I wrote the pre-filler for each page as a function, and I serially called each function at the end of the previous function, using setTimeout(nextfunct, 10000) to let the page update before executing the next line. (I was going to refine that by trying to create some kind of object listener instead of an arbitrary 10 second delay, but I wasn't even able to get that far.) This approach creates two errors.
1) The script seems to be checking whether the elements are on the DOM before the end of the setTimeout(), so it still gives me an error. If nextfunct is defined as
document.getElementById("doesntexistyet").value = "Fill Me";
console.log("nextfunct ran");
I will get the error message stating there is no element with the id "doesntexistyet" immediately, not after a delay of 10 seconds. The element on the next page will not update.
2) The DOM updating interrupts my script. In the above code, the console output will not ever appear in my console. If I comment out the missing element, so the function only prints a comment, it will still not appear in my console. However, if I comment out the code and I switch the setTimeout to 1ms, "nextfunct ran" will appear in my console, until the page updates, at which time the console will be deleted.
Are there ways around this which I can implement using only vanilla JS and a browser? I'm sure there's a keyword I can search for where someone has discussed this before, but it seems like the vast majority of JS autofilling discussions are oriented towards people designing code to be integrated into a website,
Thanks
I am doing GUI automation of my website via Selenium (RobotFramework). The problem I am facing is:
When my automation script clicks on some element (button,link etc) which is supposed to perform some action, it dose nothing. This happening randomly. When we test is manually, it works all the time. One observation is, with slower machine/environment, this tends to happen more.
I suspect this is happening either due to some corresponding JS is not loaded yet or if there is any such thing called "action binding" with each elemetnt, has not happened.
Some question
- Is there a way to find out if all the JS calls are over?
- In case action binding happens, has it already bound or not.
Please share if you have any other solution.
do you know what is last to load on the page? This should be very easy to find out via Developer Tools in your browser of choice.
You can then easily use a Wait Until Keyword (there are many variations) to wait until that last item appears as you expect, then continue with your test. Setting the timeout length and interval will help control the overhead of time/performance.
Wait Until Element Is Visible id=finalElement 10 finalElement did not appear on the screen before timeout period
http://robotframework.org/Selenium2Library/Selenium2Library.html - please see the documentation for further examples and options in terms of keywords
at the moment I am working on replacing pop-up on a website I inherited. Those pop-ups used modal dialogs, which are on their way out and even were dropped by pop-up blockers on client side.
My approach was loading the HTML of the pop-up into an div on the main site, while hiding the original content, then emptying the div and switch the main content back to visible.
This works so fa, but the loaded content has scripts that run to check if someone is already using that function. The first time you use that function all is fine. The script runs, sees noone is using the function, I go through the stuff, empty the div and return to the main content. When trying to use the function a second time the script to still run (console shows the requests), even though I emptied the div, prompting the eternal "please wait till other user is finished" lines, since the first script is still checking for use, signalling the second script "I'm buisy".
So right now, I am looking for a way to stop the first script, since removing the HTML-content doesn't suffice it seems.
The code so far:
$("#dialog").load("stuffToLoad.htm",function(response, status)
{
if(status=="success"){
$(".fullTable").toggle();
$("#dialog").toggle();
};
})
to get the content. The in-use-check is done with a post-request, that is repeated by a window.setTimeout every second. That part seems to still run.
When everything is done, or the user runs into an error I tried:
function returnToProcVal()
{
$("#dialog").html("");
$("#dialog").toggle();
$(".fullTable").toggle();
}
to delete the function content and scripts, to stop them running. While the DOM says all is gone I can see the post requests being repeated in the console.
I'd be grateful for any pointer or perhaps even better methods to get the function running and returning the user, without the use of pop-ups.
I am writing some automated tests in Selenium 2.0 using the Firefox driver. The site uses many bindings, for example an 'input' tag with a 'data-val-method-to-execute' attribute which triggers a javascript function.
Some 10% of my tests fail randomly because the driver clicks an element before the corresponding javascript function got loaded - so nothing will happen.
One solution could be Thread.Sleep - problems are: I would have to implement in all of my tests(quite a lot). They will slow my tests drastically, and time is an issue. A simple dropdown with a sleep of 1 second doesn't always work (so I would have to increase the timeout)
You need to figure out a way to make Selenium wait until the page is ready. There is no "one size fits all" solution for this. It really depends on what you do.
For some complex AJAX/JavaScript, I had to use phantomjs and add code which counts the number of open/active network connections. The test would wait until the number changes (so I know the AJAX request has been sent) and then until the number of active connections drops back to 0 (so I know the AJAX is completed).
Alternatively, try to add a hidden DIV to the page which tells the test "all scripts have finished". Wait for the DIV to appear. The problem here is to make sure that your AJAX handlers create the DIV.
In order to avoid polluting your code, use an empty function which creates the DIV and insert an additional <script> element in the head when running the tests which overwrites the function.