This question already has answers here:
How to pass a function in Puppeteers .evaluate() method
(5 answers)
Closed 5 months ago.
I am using Puppeteer for headless Chrome. I wish to evaluate a function inside the page that uses parts of other functions, defined dynamically elsewhere.
The code below is a minimal example / proof. In reality functionToInject() and otherFunctionToInject() are more complex and require the pages DOM.
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto(someURL);
var functionToInject = function(){
return 1+1;
}
var otherFunctionToInject = function(input){
return 6
}
var data = await page.evaluate(function(functionToInject, otherFunctionToInject){
console.log('woo I run inside a browser')
return functionToInject() + otherFunctionToInject();
});
return data
When I run the code, I get:
Error: Evaluation failed: TypeError: functionToInject is not a function
Which I understand: functionToInject isn't being passed into the page's JS context. But how do I pass it into the page's JS context?
You can add function to page context with addScriptTag:
const browser = await puppeteer.launch();
const page = await browser.newPage();
function functionToInject (){
return 1+1;
}
function otherFunctionToInject(input){
return 6
}
await page.addScriptTag({ content: `${functionToInject} ${otherFunctionToInject}`});
var data = await page.evaluate(function(){
console.log('woo I run inside a browser')
return functionToInject() + otherFunctionToInject();
});
console.log(data);
await browser.close();
This example is a dirty way of solving this problem with string concatenation. More clean would be using a url or path in the addScriptTag method.
Or use exposeFunction (but now functions are wrapped in Promise):
const browser = await puppeteer.launch();
const page = await browser.newPage();
var functionToInject = function(){
return 1+1;
}
var otherFunctionToInject = function(input){
return 6
}
await page.exposeFunction('functionToInject', functionToInject);
await page.exposeFunction('otherFunctionToInject', otherFunctionToInject);
var data = await page.evaluate(async function(){
console.log('woo I run inside a browser')
return await functionToInject() + await otherFunctionToInject();
});
console.log(data);
await browser.close();
working example accessible by link, in the same repo you can see the tested component.
it("click should return option value", async () => {
const optionToReturn = "ClickedOption";
const page = await newE2EPage();
const mockCallBack = jest.fn();
await page.setContent(
`<list-option option='${optionToReturn}'></list-option>`
);
await page.exposeFunction("functionToInject", mockCallBack); // Inject function
await page.$eval("list-option", (elm: any) => {
elm.onOptionSelected = this.functionToInject; // Assign function
});
await page.waitForChanges();
const element = await page.find("list-option");
await element.click();
expect(mockCallBack.mock.calls.length).toEqual(1); // Check calls
expect(mockCallBack.mock.calls[0][0]).toBe(optionToReturn); // Check argument
});
You can also use page.exposeFunction() which will make your function return a Promise (requiring the use of async and await). This happens because your function will not be running inside your browser, but inside your nodejs application and its results are being send back and forth into/to the browser code.
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto(someURL);
var functionToInject = function(){
return 1+1;
}
var otherFunctionToInject = function(input){
return 6
}
await page.exposeFunction("functionToInject", functionToInject)
await page.exposeFunction("otherFunctionToInject", otherFunctionToInject)
var data = await page.evaluate(async function(){
console.log('woo I run inside a browser')
return await functionToInject() + await otherFunctionToInject();
});
return data
Related questions:
exposeFunction() does not work after goto()
exposed function queryseldtcor not working in puppeteer
How to use evaluateOnNewDocument and exposeFunction?
exposeFunction remains in memory?
Puppeteer: pass variable in .evaluate()
Puppeteer evaluate function
allow to pass a parameterized funciton as a string to page.evaluate
Functions bound with page.exposeFunction() produce unhandled promise rejections
How to pass a function in Puppeteers .evaluate() method?
Why can't I access 'window' in an exposeFunction() function with Puppeteer?
Related
So I just started learning puppeteer.js and what happens is that my code runs it runs without bugs but it does not display anything to the console. (I use Node.js for debugging purposes) The only time is does display something is when I put the async function inside another function and even then it return undefined. I was wondering why this is and how to fix it.
Here is my code:
(async () =>{
let movieUrl = 'https://www.imdb.com/title/tt0111161/?ref_=nav_sr_1'
let browser = await puppeteer.launch();
let page = await browser.newPage();
await page.goto(movieUrl, {waitUntil: 'networkidle2'})
let data = await page.evaluate(() => {
let title = document.querySelector('div[class="title_wrapper"] > h1').innerText;
let rating = document.querySelector('span[itemprop="ratingValue"]').innerText;
let ratingCount = document.querySelector('span[itemprop="ratingCount"]').innerText;
return{title, rating, ratingCount};
})
console.log(data);
debugger;
await browser.close();
})
This is an unnamed async function:
async () => {
// the function code
}
If you want to run it straight away, you need to call it. You can do it by enclosing it in parentheses:
(async () => {
// the function code
})()
This is called IIFE, Immediately-invoked Function Expression. You declare a function and run it immediately.
See these articles to learn about it:
https://developer.mozilla.org/en-US/docs/Glossary/IIFE
https://flaviocopes.com/javascript-iife/
Excuse me, but the documentation is a little incomprehensible to me.
I use the argument:
const myDiv = await page.$$eval(".myDiv", myDiv => myDiv.textContent);
but the console.log will only return one result while the results for this div are >10.
How do I display them all ?
edit// That's my code that I'm learning from
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('mypage');
// await page.screenshot({path: 'example.png'});
await page.waitForSelector(".myDiv");
const myDiv = await page.$eval(".myDiv", myDiv => myDiv.textContent);
console.log(myDiv);
await browser.close();
})();
You can use page.evaluate:
const myDiv = await page.evaluate(() => {
const divs = Array.from(document.querySelectorAll('.myDiv'))
return divs.map(d => d.textContent)
});
Function passed to page.evaluate will be serialized and sent to browser, so it is executed in browser context (not Node).
Since you did not provide more code, this answer is pretty opinionated and maybe doesn't solve your problem. But it shows you a way how to understand what is happening.
Exspecially in developement, it's very helpful to use a combination of page.exposeFunction() and page.evaluate() to see what is going on in the browser and also in node/puppeteer.
Here is a draft which I hope it helps you to understand.
(async () => {
function executedInNodeContext(result) {
//this prints in the Node Console
console.log(result);
}
function executedInBrowserContext() {
console.log('Im in the Browser');
const myDiv = [...document.querySelectorAll('.myDiv')];
window.nameOfNodeFunction(myDiv);
}
// See the browser
const browser = await puppeteer.launch({ headless: false });
// Create a new page
const page = await browser.newPage();
// Callback in Node Context
await page.exposeFunction('nameOfNodeFunction', executedInNodeContext);
// Send puppeteer to a Url
await page.goto('http://localhost/awesome/URL');
// Function executed in the Browser on the given page
await page.evaluate(executedInBrowserContext);
})();
page.$$eval() sends in its callback an array with elements, so you need something like this to get all elements data:
const myDivs = await page.$$eval(".myDiv", divs => divs.map(div => div.textContent));
How do I execute client-side JS code within the page.evaluate() statement (not just browser JavaScript code, Node.js code)?
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://example.com');
await page.screenshot({path: 'example.png'});
await page.evaluate(() => {
document.querySelector('button[type=submit]').click();
});
console.log('yes')
await browser.close();
})();
The first parameter passed to page.evaluate() should be a function which will be evaluated in the page context in the browser.
Node.js is server-side code, and is meant to be executed on the server.
You can pass arguments from the Node.js environment to the page function using the following method:
// Node.js Environment
const hello_world = 'Hello, world! (from Node.js)';
await page.evaluate(hello_world => {
// Browser Page Environment
console.log(hello_world);
}, hello_world);
You can listen for the 'console' event to occur in the page context and print the result using page.on():
page.on('console', msg => {
for (let i = 0; i < msg.args().length; i++) {
console.log(`${i}: ${msg.args()[i]}`);
}
});
I am trying to do some scraping using a library and my code uses Node's
async/await pattern.
I have defined a variable 'page' in function named 'sayhi' and I pass the same variable to function ex, I get error while running the code.
const puppeteer = require('puppeteer');
async function sayhi() {
const browser = await puppeteer.launch({headless: false});
const page = await browser.newPage();
await page.goto('https://www.example.com/'); //
ex(page); //FAILS
var frames2 = await newpage.frames(); // WORKS
}
function ex(newpage){
var frames = await newpage.frames(); // FAILING
}
sayhi();
You're using await in a function that isn't an async function. Try this instead:
async function ex(newpage) {
If you need frames2 to run only after ex is finished completely, you'll also want to await ex(page); in sayhi.
I can make a change to a property and trigger a re-render of an element on this page via the console
Open JBrowse demo site via this link then read on...
In console I can run the following to update 1 element (in reality I'd do them all):
document.querySelectorAll('.track_jbrowse_view_track_alignments2')[0].track.displayMode = 'compact'
document.querySelectorAll('.track_jbrowse_view_track_alignments2')[0].track.layout = null
document.querySelectorAll('.track_jbrowse_view_track_alignments2')[0].track.redraw()
I'm attempting to perform this in the puppeteer code with:
const tracks = await page.$$('.track_jbrowse_view_track_alignments2');
for (let t of tracks) {
await page.evaluate(t => {
t.displayMode = 'compact';
t.layout = null;
t.redraw();
}, t);
}
The existing functional script is under this link, the above snippet would be inserted immediately following the highlighted line.
Any guidance would be great, thanks.
You should wait for your JS code when it is fully ready. It's up to you how you detect it, I just use dummy await page.waitFor(10000);. Next, you should invoke your operation on t.track object not on t.
Here is working example:
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto(
'http://jbrowse.org/code/JBrowse-1.12.4/?loc=ctgA%3A16801..23754&tracks=volvox-sorted_bam&data=sample_data%2Fjson%2Fvolvox&nav=0&tracklist=0&fullviewlink=0&highlight='
);
await page.waitFor(10000);
const tracks = await page.$$('.track_jbrowse_view_track_alignments2');
for (let t of tracks) {
await page.evaluate(t => {
t.track.displayMode = 'compact';
t.track.layout = null;
t.track.redraw();
}, t);
}
await page.screenshot({ path: 'image.png' });
await browser.close();
})();