Related
So I have an line which I can just paste manually into the Devtools Console in a browser. Is there any way to make pupeteer execute it? After searching I havent found anything, sorry if this has been answered already, I am quite new.
For those who care its an Line to buy an listing of an Item, Example:
BuyMarketListing('listing', '3555030760772417847', 730, '2', '24716958303')
It looks like you're looking for page.evaluate(). Here is a link to the Puppeteer's documentation for it. You can pass in a string or an anonymous function containing the lines you want to evaluate in the page.
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://example.com');
await page.evaluate(() => { insert lines here }); // page.evaluate() should run the lines in the browser console
await browser.close();
})();
I submit a form using the following code and i want Puppeteer to wait page load after form submit.
await page.click("button[type=submit]");
//how to wait until the new page loads before taking screenshot?
// i don't want this:
// await page.waitFor(1*1000); //← unwanted workaround
await page.screenshot({path: 'example.png'});
How to wait for page load with puppeteer?
You can wait for navigation asynchronously to avoid getting null on redirection,
await Promise.all([
page.click('button[type=submit]'),
page.waitForNavigation({waitUntil: 'networkidle2'})
]);
This will help you if the page.click already triggers a navigation.
await page.waitForNavigation();
According to the Official Documentation, you should use:
page.waitForNavigation(options)
options <Object> Navigation parameters which might have the following properties:
timeout <number> Maximum navigation time in milliseconds, defaults to 30 seconds, pass 0 to disable timeout. The default value can be changed by using the page.setDefaultNavigationTimeout(timeout) method.
waitUntil <string|Array<string>> When to consider navigation succeeded, defaults to load. Given an array of event strings, navigation is considered to be successful after all events have been fired. Events can be either:
load - consider navigation to be finished when the load event is fired.
domcontentloaded - consider navigation to be finished when the DOMContentLoaded event is fired.
networkidle0 - consider navigation to be finished when there are no more than 0 network connections for at least 500 ms.
networkidle2 - consider navigation to be finished when there are no more than 2 network connections for at least 500 ms.
returns: <Promise<[?Response]>> Promise which resolves to the main resource response. In case of multiple redirects, the navigation will resolve with the response of the last redirect. In case of navigation to a different anchor or navigation due to History API usage, the navigation will resolve with null.
Readability:
You can use page.waitForNavigation() to wait for a page to navigate:
await page.waitForNavigation();
Performance:
But since page.waitForNavigation() is a shortcut for page.mainFrame().waitForNavigation(), we can use the following for a minor performance enhancement:
await page._frameManager._mainFrame.waitForNavigation();
Sometimes even using await page.waitForNavigation() will still result in a Error: Execution context was destroyed, most likely because of a navigation.
In my case, it was because the page was redirecting multiple times. The API says the default waitUntil option is Load—this required me to wait for navigation each redirect (3 times).
Using only a single instance of page.waitForNavigation with the waitUntil option networkidle2 worked well in my case:
await button.click();
await page.waitForNavigation({waitUntil: 'networkidle2'});
Finally, the API suggests using a Promise.All to prevent a race condition. I haven't needed this but provide it for completeness:
await Promise.all([button.click(), page.waitForNavigation({waitUntil:'networkidle2'})])
If all else fails, you can use page.waitForSelector as recommended on a Puppeteer github issue—or in my case, page.waitForXPath()
I know it is bit late to answer this. It may be helpful for those who are getting below exception while doing waitForNavigation.
(node:14531) UnhandledPromiseRejectionWarning: TimeoutError:
Navigation Timeout Exceeded: 30000ms exceeded
at Promise.then (/home/user/nodejs/node_modules/puppeteer/lib/LifecycleWatcher.js:142:21)
at -- ASYNC --
at Frame. (/home/user/nodejs/node_modules/puppeteer/lib/helper.js:111:15)
at Page.waitForNavigation (/home/user/nodejs/node_modules/puppeteer/lib/Page.js:649:49)
at Page. (/home/user/nodejs/node_modules/puppeteer/lib/helper.js:112:23)
at /home/user/nodejs/user/puppeteer/example7.js:14:12
at
The correct code that worked for me is as below.
await page.click('button[id=start]', {waitUntil: 'domcontentloaded'});
Similarly if you are going to a new page, code should be like
await page.goto('here goes url', {waitUntil: 'domcontentloaded'});
i suggest to wrap page.to in a wrapper and wait for everything loaded
this is my wrapper
loadUrl: async function (page, url) {
try {
await page.goto(url, {
timeout: 20000,
waitUntil: ['load', 'domcontentloaded', 'networkidle0', 'networkidle2']
})
} catch (error) {
throw new Error("url " + url + " url not loaded -> " + error)
}
}
now you can use this with
await loadUrl(page, "https://www.google.com")
None of the above answers solved my issue. Sometimes waitForNavigation just timeout. I came up with other solution using the waitForFunction, checking if document is in ready state.
await page.waitForFunction(() => document.readyState === "complete");
await Promise.all([
page.click(selectors.submit),
page.waitForNavigation({ waitUntil: 'networkidle0' }),
]);
This would be the first priority to use as it waits for all network to complete and assumes it is done when you don't have more than 0 network call for 500ms.
you can also use
await page.waitForNavigation({ waitUntil: 'load' })
or else, you can use
await page.waitForResponse(response => response.ok())
this function can also be used in various places as it only allows to proceed further when all the calls are a success that is when all the response status is ok i.e (200-299)
This worked for me:
await Promise.all([
page.goto(URL),
page.waitForNavigation({ waitUntil: 'networkidle0' }),
]);
console.log('page loaded')
For some reason I was not able to click button (Handled an event, not in form)
<button onclick="someFunction();" class="button button2">Submit</button>
The problem was that page was rendered on server side. Thus the button didn't existed whenever I waited for input field await page.waitForSelector('button.button2')
The solution was to bind page.goto(URL) and page.waitForNavigation({ waitUntil: 'networkidle0' }) in Promise
await Promise.all([
page.goto(URL),
page.waitForNavigation({ waitUntil: 'networkidle0' }),
]);
console.log('page loaded')
await page.waitForSelector('button.button2')
console.log('button is here');
If submitting the form opens some other page, then you may just want to wait for a selector in that page. I have often had issues using page.waitForNavigation() since it's options don't really ensure we have effectively navigated to another page.
// login page
page.click("#login");
// homepage, after login
page.waitForSelector("#home", {visible: true}); // page.waitForXpath()
Of you course you can increase the wait time for the selector.
This works for me
Puppeteer version: 19.2.2
page.click(".clickable-selector");
await page.waitForNavigation({ waitUntil: "load" });
Note:
If you do this inside a loop. ( scrapping page-1, click to page-2, scrapping page-2 and so on... )
await page.waitForSelector(".clickable-selector", { visible: true });
Wait for this clickable selector before doing any other scrapping on the page.
I ran into a scenario, where there was the classic POST-303-GET and an input[type=submit] was involved. It seems that in this case, the click of the button won't resolve until after the associated form's submission and redirection, so the solution was to remove the waitForNavigation, because it was executed after the redirection and thus was timing out.
Please try
await page.waitForNavigation()
or
await page.waitForSelector("#indecator_of_any_element_of_you_are_waiting_for")
I'm new to puppeteer and I'm trying to click on a selector from a dropdown menu the MR element here
I've tried using await page.click('.mat-option ng-star-inserted mat-active');
and also
await page.select('#mat-option-0');
here is my code, would anyone be able to help me fix this issue and understand how to resolve it in the future? I'm not to sure what methods to be using with each elelement, I think it's every time I introduce a class with spaces in the name could that be the issue?
and does anyone have any best practices for when codings things like this?
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();
await page.goto('https://www.game.co.uk/en/-2640058?cm_sp=NintendoFormatHub-_-Accessories-_-espot-_-PikaCase');
await console.log('Users navigated to site :)');
await page.waitFor(2300);
await page.click('.cookiePolicy_inner--actions');
await page.waitFor(1000);
await page.click('.addToBasket');
await page.waitFor(1300);
await page.click('.secure-checkout');
await page.waitFor(2350);
await page.click('.cta-large');
await page.waitFor(1200);
await page.goto('https://checkout.game.co.uk/contact');
await page.waitFor(500);
await page.click('.mat-form-field-infix');
await page.waitForSelector('.ng-tns-c17-1 ng-trigger ng-trigger-transformPanel mat-select-panel mat-primary');
await page.click('.mat-option ng-star-inserted mat-active');
})();
There are a couple of issues with the script, let's see them:
you are using waitFor() with a number of miliseconds, this is brittle because you never know if perhaps some action will take longer, and if it does not, you will waste time; you can substitute these waits with waitForSelector(); in fact, if you use VSCode (and perhaps other IDEs), it will notify you that this method is deprecated, don't ignore these warnings:
when I use DevTools, no element is returned for .mat-option ng-star-inserted mat-active selector, but I can find the desired element with #mat-option-0 selector, or I can use the longer version, but have to use a dot (.) before each class and delete spaces between them like so .mat-option.ng-star-inserted.mat-active, you can see a CSS reference here, the point is that with spaces, it looks for descendants, which is not what you want
These two changes should give you what you need, this is a result when running on my side, you can see that Mr. has been selected:
I got there with this script:
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();
await page.goto('https://www.game.co.uk/en/-2640058?cm_sp=NintendoFormatHub-_-Accessories-_-espot-_-PikaCase');
await console.log('Users navigated to site :)');
await page.waitForSelector('.cookiePolicy_inner--actions');
await page.click('.cookiePolicy_inner--actions');
await page.waitForSelector('.addToBasket');
await page.click('.addToBasket');
await page.waitForSelector('.secure-checkout');
await page.click('.secure-checkout');
await page.waitForSelector('.cta-large');
await page.click('.cta-large');
await page.goto('https://checkout.game.co.uk/contact');
await page.waitForSelector('.mat-form-field-infix');
await page.click('.mat-form-field-infix');
await page.waitForSelector('#mat-option-0');
await page.click('#mat-option-0');
})();
However, this is still not ideal because:
you handle the cookie bar with clicks, try to find a way without clicking; perhaps injecting a cookie that disables the cookie bar (if possible)
the code is one big piece that is perhaps ok for now and this example but might become unmaintainable if you keep adding lines to it; try to reuse code in functions and methods
I'm practicing with Telegram bots and puppeteer so I've decided to create a bot to order pizza from a specific website.
Once the bot has taken the order he needs to place the data he took to inputs on the page, here how it looks like:
These two fields are spans and when puppeteer clicks on the enabled one (left) he gets an input to complete. Then when the first input is done puppeteer has to do the exact same procedure with the second field: click on <span> tag, place data in input, etc.
But the thing is that there is a small-time gap between the completion of the first field and activation of the second one. My bot doesn't recognize this gap and clicks on the second field's span instantly (and of course it doesn't work).
Here's a code fragment:
await page.waitForXPath('//*[#id="select2-chosen-2"]', {visible: true})
const [secondSpan] = await page.$x('//*[#id="select2-chosen-2"]')
await secondSpan.click()
When I type node bot with this fragment I get no errors or warnings. But as I said it takes some time for the second field to activate. I've found a function to make puppeeter stop the execution of my code for a certain time period: page.waitForTimeout().
Here the example of usage in puppeteer's documentation:
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
page.waitForTimeout(1000).then(() => console.log('Waited a second!'));
await browser.close();
})();
https://pptr.dev/#?product=Puppeteer&version=v10.1.0&show=api-pagewaitfortimeoutmilliseconds
Here's my case:
await page.waitForXPath('//*[#id="select2-chosen-2"]', {visible: true})
const [secondSpan] = await page.$x('//*[#id="select2-chosen-2"]')
page.waitForTimeout(1500)
await secondSpan.click()
This code also doesn't show any error, but it also doesn't click on the field. When I add await to page.waitForTimeout() I get this error:
Error: Node is either not visible or not an HTMLElement
How can I fix it?
So all I needed was to put this code:
await page.click('#s2id_home-number-modal')
Or using XPath:
const [secondSpan] = await page.$x('//*[#id="select2-chosen-2"]')
await secondSpan.click()
into .then() method, that is called after page.setTimeout(500).
All in all, it looks like this (by the way, I've changed some selectors, but it's not a big deal):
await page.waitForTimeout(500).then(async () => {
await page.click('#s2id_home-number-modal')
})
I submit a form using the following code and i want Puppeteer to wait page load after form submit.
await page.click("button[type=submit]");
//how to wait until the new page loads before taking screenshot?
// i don't want this:
// await page.waitFor(1*1000); //← unwanted workaround
await page.screenshot({path: 'example.png'});
How to wait for page load with puppeteer?
You can wait for navigation asynchronously to avoid getting null on redirection,
await Promise.all([
page.click('button[type=submit]'),
page.waitForNavigation({waitUntil: 'networkidle2'})
]);
This will help you if the page.click already triggers a navigation.
await page.waitForNavigation();
According to the Official Documentation, you should use:
page.waitForNavigation(options)
options <Object> Navigation parameters which might have the following properties:
timeout <number> Maximum navigation time in milliseconds, defaults to 30 seconds, pass 0 to disable timeout. The default value can be changed by using the page.setDefaultNavigationTimeout(timeout) method.
waitUntil <string|Array<string>> When to consider navigation succeeded, defaults to load. Given an array of event strings, navigation is considered to be successful after all events have been fired. Events can be either:
load - consider navigation to be finished when the load event is fired.
domcontentloaded - consider navigation to be finished when the DOMContentLoaded event is fired.
networkidle0 - consider navigation to be finished when there are no more than 0 network connections for at least 500 ms.
networkidle2 - consider navigation to be finished when there are no more than 2 network connections for at least 500 ms.
returns: <Promise<[?Response]>> Promise which resolves to the main resource response. In case of multiple redirects, the navigation will resolve with the response of the last redirect. In case of navigation to a different anchor or navigation due to History API usage, the navigation will resolve with null.
Readability:
You can use page.waitForNavigation() to wait for a page to navigate:
await page.waitForNavigation();
Performance:
But since page.waitForNavigation() is a shortcut for page.mainFrame().waitForNavigation(), we can use the following for a minor performance enhancement:
await page._frameManager._mainFrame.waitForNavigation();
Sometimes even using await page.waitForNavigation() will still result in a Error: Execution context was destroyed, most likely because of a navigation.
In my case, it was because the page was redirecting multiple times. The API says the default waitUntil option is Load—this required me to wait for navigation each redirect (3 times).
Using only a single instance of page.waitForNavigation with the waitUntil option networkidle2 worked well in my case:
await button.click();
await page.waitForNavigation({waitUntil: 'networkidle2'});
Finally, the API suggests using a Promise.All to prevent a race condition. I haven't needed this but provide it for completeness:
await Promise.all([button.click(), page.waitForNavigation({waitUntil:'networkidle2'})])
If all else fails, you can use page.waitForSelector as recommended on a Puppeteer github issue—or in my case, page.waitForXPath()
I know it is bit late to answer this. It may be helpful for those who are getting below exception while doing waitForNavigation.
(node:14531) UnhandledPromiseRejectionWarning: TimeoutError:
Navigation Timeout Exceeded: 30000ms exceeded
at Promise.then (/home/user/nodejs/node_modules/puppeteer/lib/LifecycleWatcher.js:142:21)
at -- ASYNC --
at Frame. (/home/user/nodejs/node_modules/puppeteer/lib/helper.js:111:15)
at Page.waitForNavigation (/home/user/nodejs/node_modules/puppeteer/lib/Page.js:649:49)
at Page. (/home/user/nodejs/node_modules/puppeteer/lib/helper.js:112:23)
at /home/user/nodejs/user/puppeteer/example7.js:14:12
at
The correct code that worked for me is as below.
await page.click('button[id=start]', {waitUntil: 'domcontentloaded'});
Similarly if you are going to a new page, code should be like
await page.goto('here goes url', {waitUntil: 'domcontentloaded'});
i suggest to wrap page.to in a wrapper and wait for everything loaded
this is my wrapper
loadUrl: async function (page, url) {
try {
await page.goto(url, {
timeout: 20000,
waitUntil: ['load', 'domcontentloaded', 'networkidle0', 'networkidle2']
})
} catch (error) {
throw new Error("url " + url + " url not loaded -> " + error)
}
}
now you can use this with
await loadUrl(page, "https://www.google.com")
None of the above answers solved my issue. Sometimes waitForNavigation just timeout. I came up with other solution using the waitForFunction, checking if document is in ready state.
await page.waitForFunction(() => document.readyState === "complete");
await Promise.all([
page.click(selectors.submit),
page.waitForNavigation({ waitUntil: 'networkidle0' }),
]);
This would be the first priority to use as it waits for all network to complete and assumes it is done when you don't have more than 0 network call for 500ms.
you can also use
await page.waitForNavigation({ waitUntil: 'load' })
or else, you can use
await page.waitForResponse(response => response.ok())
this function can also be used in various places as it only allows to proceed further when all the calls are a success that is when all the response status is ok i.e (200-299)
This works for me
Puppeteer version: 19.2.2
page.click(".clickable-selector");
await page.waitForNavigation({ waitUntil: "load" });
Note:
If you do this inside a loop. ( scrapping page-1, click to page-2, scrapping page-2 and so on... )
await page.waitForSelector(".clickable-selector", { visible: true });
Wait for this clickable selector before doing any other scrapping on the page.
This worked for me:
await Promise.all([
page.goto(URL),
page.waitForNavigation({ waitUntil: 'networkidle0' }),
]);
console.log('page loaded')
For some reason I was not able to click button (Handled an event, not in form)
<button onclick="someFunction();" class="button button2">Submit</button>
The problem was that page was rendered on server side. Thus the button didn't existed whenever I waited for input field await page.waitForSelector('button.button2')
The solution was to bind page.goto(URL) and page.waitForNavigation({ waitUntil: 'networkidle0' }) in Promise
await Promise.all([
page.goto(URL),
page.waitForNavigation({ waitUntil: 'networkidle0' }),
]);
console.log('page loaded')
await page.waitForSelector('button.button2')
console.log('button is here');
If submitting the form opens some other page, then you may just want to wait for a selector in that page. I have often had issues using page.waitForNavigation() since it's options don't really ensure we have effectively navigated to another page.
// login page
page.click("#login");
// homepage, after login
page.waitForSelector("#home", {visible: true}); // page.waitForXpath()
Of you course you can increase the wait time for the selector.
I ran into a scenario, where there was the classic POST-303-GET and an input[type=submit] was involved. It seems that in this case, the click of the button won't resolve until after the associated form's submission and redirection, so the solution was to remove the waitForNavigation, because it was executed after the redirection and thus was timing out.
Please try
await page.waitForNavigation()
or
await page.waitForSelector("#indecator_of_any_element_of_you_are_waiting_for")