We Keep Coding

Passing functions inside $$eval or $eval func., Puppeter.js [duplicate]

Question
How do I expose an object with a bunch of methods to puppeteer? I am trying to retain the definition of the parent object and method (i.e. foo.one) within page.evaluate, if possible. In other words, I am looking for console.log(foo.one('world')), typed as such, to return world.
Background
foo is a library container which returns a whole bunch of (relatively) pure functions. These functions are required both in the main script context AND within the puppeteer browser. I would prefer to not have to redefine each of them within page.evaluate and instead pass this entire "package" to page.evaluate for repository readability/maintenance. Nonetheless, as one answer suggests below, iterating over the methods from foo and exposing them individually to puppeteer with a different name isn't a terrible option. It just would require redefinitions within page.evaluate which I am trying to avoid.
Expected vs Actual
Let's assume an immediately invoked function which returns an object with a series of function definitions as properties. When trying to pass this IIFE (or object) to puppeteer page, I receive the following error:
import puppeteer from 'puppeteer'
const foo = (()=>{
const one = (msg) => console.log('1) ' + msg)
const two = (msg) => console.log('2) ' + msg)
const three = (msg) => console.log('3) ' + msg)
return {one, two, three}
})()
const browser = await puppeteer.launch().catch(err => `Browser not launched properly: ${err}`)
const page = await browser.newPage()
page.on('console', (msg) => console.log('PUPPETEER:', msg._text)); // Pipe puppeteer console to local console
await page.evaluate((foo)=>{
console.log('hello')
console.log(foo.one('world'))
},foo)
browser.close()
// Error: Evaluation failed: TypeError: foo.one is not a function
When I try to use page.exposeFunction I receive an error. This is to be expected because foo is an object.
page.exposeFunction('foo',foo)
// Error: Failed to add page binding with name foo: [object Object] is not a function or a module with a default export.
The control case, defining the function within the browser page, works as expected:
import puppeteer from 'puppeteer'
const browser = await puppeteer.launch().catch(err => `Browser not launched properly: ${err}`)
const page = await browser.newPage()
page.on('console', (msg) => console.log('PUPPETEER:', msg._text)); // Pipe puppeteer console to local console
await page.evaluate(()=>{
const bar = (()=>{
const one = (msg) => console.log('1) ' + msg)
const two = (msg) => console.log('2) ' + msg)
const three = (msg) => console.log('3) ' + msg)
return {one, two, three}
})()
console.log('hello')
console.log(bar.one('world'))
})
browser.close()
// PUPPETEER: hello
// PUPPETEER: 1) world
Update (5/19/2022)
Adding a quick update after testing the below solutions given my use case
Reminder: I am trying to pass an externally defined utilities.js library to the browser so that it can conditionally interact with page data and navigate accordingly.
I'm open to any ideas or feedback!
addScriptTag()
Unfortunately, passing a node.js module of utility functions is very difficult in my situation. When the module contains export statements or objects, addScriptTag() fails.
I get Error: Evaluation failed: ReferenceError: {x} is not defined in this case. I created an intermediary function to remove the export statements. That is messy but it seemed to work. However, some of my functions are IIFE which return an object with methods. And objects are proving very hard to work with via addScriptTag(), to say the least.
redundant code
I think for smaller projects the simplest and best option is to just re-declare the objects/functions in the puppeteer context. I hate redefining things but it works as expected.
import()
As #ggorlen suggests, I was able to host the utilities function on another server. This can be sourced by both the node.js and puppeteer environments. I still had to import the library twice: once in the node.js environment and once in the browser context. But it's probably better in my case than redeclaring dozens of functions and objects.

It might be repetitive when calling, but you could iterate over the object and call page.exposeFunction for each.
page.exposeFunction('fooOne', foo.one);
// ...
or
for (const [fnName, fn] of Object.entries(foo)) {
page.exposeFunction(fnName, fn);
}
If the functions can all be executed in the context of the page, simply defining them inside a page.evaluate would work too.
page.evaluate(() => {
window.foo = (()=>{
const one = (msg) => console.log('1) ' + msg)
const two = (msg) => console.log('2) ' + msg)
const three = (msg) => console.log('3) ' + msg)
return {one, two, three}
})();
});
If you have to have only a single object containing the functions in the context of the page, you could first put an object on the window with page.evaluate, then in the main script, have an serial async loop over the keys and values of the object that:
calls page.exposeFunction('fnToMove' with the function
calls page.evaluate which assigns fnToMove to the desired property on the object created earlier
But that's somewhat convoluted. I wouldn't recommend it unless you really need it.

This is a bit speculative, because the use case matters quite a bit here. For example, exposeFunction means the code runs in Node context, so that involves inter-process communication and data serialization and deserialization, which seems inappropriate for your use case of processing the data fully in the browser. Then again, if there are Node-specific tasks like reading files or making cross-origin requests, it's appropriate.
If, on the other hand, you want to add code for the browser to call in the console context, a scalable way is to put your library into a script, then use page.addScriptTag("./your-lib.js") to attach it to the window. Either use a bundler to build the lib for browser compatibility or attach it by hand. Use module.exports if you also want to import it in Node.
For example:
foo.js
;(function () {
var foo = {
one: function () { return 1; },
two: function () { return 2; },
// ...
};
if (typeof module === "object" &&
typeof module.exports === "object") {
module.exports = foo;
}
if (typeof window === "object") {
window.foo = foo;
}
})();
foo-tester.js
const puppeteer = require("puppeteer"); // ^13.5.1
const foo = require("./foo"); // also use it in Node if you want...
let browser;
(async () => {
browser = await puppeteer.launch({headless: true});
const [page] = await browser.pages();
await page.addScriptTag({path: "./foo.js"});
console.log(foo.one()); // => 1
console.log(await page.evaluate(() => foo.two())); // => 2
})()
.catch(err => console.error(err))
.finally(() => browser?.close())
;
addScriptTag also works for modules and raw JS strings. For example, this works too:
await page.addScriptTag({content: `
window.foo = {two() { return 2; }};
`});
console.log(await page.evaluate(() => foo.two())); // => 2
A hacky approach is to stringify the object of functions you may have in Node. I don't recommend this, but it's possible:
const foo = {
one() { return 1; },
two() { return 2; },
};
const fooToWindow = `window.foo = {
${Object.values(foo).map(fn => fn.toString())}
}`;
await page.addScriptTag({content: fooToWindow});
console.log(await page.evaluate(() => foo.two())); // => 2
See also Is there a way to use a class inside Puppeteer evaluate?

How can I execute some async tasks in parallel with limit in generator function?

I'm trying to execute some async tasks in parallel with a limitation on the maximum number of simultaneously running tasks.
There's an example of what I want to achieve:
Currently this tasks are running one after another. It's implemented this way:
export function signData(dataItem) {
cadesplugin.async_spawn(async function* (args) {
//... nestedArgs assignment logic ...
for (const id of dataItem.identifiers) {
yield* idHandler(dataItem, id, args, nestedArgs);
}
// some extra logic after all tasks were finished
}, firstArg, secondArg);
}
async function* idHandler(edsItem, researchId, args, nestedArgs) {
...
let oDocumentNameAttr = yield cadesplugin.CreateObjectAsync("CADESCOM.CPAttribute");
yield oDocumentNameAttr.propset_Value("Document Name");
...
// this function mutates some external data, making API calls and returns void
}
Unfortunately, I can't make any changes in cadesplugin.* functions, but I can use any external libraries (or built-in Promise) in my code.
I found some methods (eachLimit and parallelLimit) in async library that might work for me and an answer that shows how to deal with it.
But there are still two problems I can't solve:
How can I pass main params into nested function?
Main function is a generator function, so I still need to work with yield expressions in main and nested functions
There's a link to cadesplugin.* source code, where you can find async_spawn (and another cadesplugin.*) function that used in my code.
That's the code I tried with no luck:
await forEachLimit(dataItem.identifiers, 5, yield* async function* (researchId, callback) {
//... nested function code
});
It leads to Object is not async iterable error.
Another attempt:
let functionArray = [];
dataItem.identifiers.forEach(researchId => {
functionArray.push(researchIdHandler(dataItem, id, args, nestedArgs))
});
await parallelLimit(functionArray, 5);
It just does nothing.
Сan I somehow solve this problem, or the generator functions won't allow me to do this?

square peg, round hole
You cannot use async iterables for this problem. It is the nature of for await .. of to run in series. await blocks and the loop will not continue until the awaited promise has resovled. You need a more precise level of control where you can enforce these specific requirements.
To start, we have a mock myJob that simulates a long computation. More than likely this will be a network request to some API in your app -
// any asynchronous task
const myJob = x =>
sleep(rand(5000)).then(_ => x * 10)
Using Pool defined in this Q&A, we instantiate Pool(size=4) where size is the number of concurrent threads to run -
const pool = new Pool(4)
For ergonomics, I added a run method to the Pool class, making it easier to wrap and run jobs -
class Pool {
constructor (size) ...
open () ...
deferNow () ...
deferStacked () ...
// added method
async run (t) {
const close = await this.open()
return t().then(close)
}
}
Now we need to write an effect that uses our pool to run myJob. Here you will also decide what to do with the result. Note the promise must be wrapped in a thunk otherwise pool cannot control when it begins -
async function myEffect(x) {
// run the job with the pool
const r = await pool.run(_ => myJob(x))
// do something with the result
const s = document.createTextNode(`${r}\n`)
document.body.appendChild(s)
// return a value, if you want
return r
}
Now run everything by mapping myEffect over your list of inputs. In our example myEffect we return r which means the result is also available after all results are fetched. This optional but demonstrates how program knows when everything is done -
Promise.all([1,2,3,4,5,6,7,8,9,10,11,12].map(myEffect))
.then(JSON.stringify)
.then(console.log, console.error)
full program demo
In the functioning demo below, I condensed the definitions so we can see them all at once. Run the program to verify the result in your own browser -
class Pool {
constructor (size = 4) { Object.assign(this, { pool: new Set, stack: [], size }) }
open () { return this.pool.size < this.size ? this.deferNow() : this.deferStacked() }
async run (t) { const close = await this.open(); return t().then(close) }
deferNow () { const [t, close] = thread(); const p = t.then(_ => this.pool.delete(p)).then(_ => this.stack.length && this.stack.pop().close()); this.pool.add(p); return close }
deferStacked () { const [t, close] = thread(); this.stack.push({ close }); return t.then(_ => this.deferNow()) }
}
const rand = x => Math.random() * x
const effect = f => x => (f(x), x)
const thread = close => [new Promise(r => { close = effect(r) }), close]
const sleep = ms => new Promise(r => setTimeout(r, ms))
const myJob = x =>
sleep(rand(5000)).then(_ => x * 10)
async function myEffect(x) {
const r = await pool.run(_ => myJob(x))
const s = document.createTextNode(`${r}\n`)
document.body.appendChild(s)
return r
}
const pool = new Pool(4)
Promise.all([1,2,3,4,5,6,7,8,9,10,11,12].map(myEffect))
.then(JSON.stringify)
.then(console.log, console.error)
slow it down
Pool above runs concurrent jobs as quickly as possible. You may also be interested in throttle which is also introduced in the original post. Instead of making Pool more complex, we can wrap our jobs using throttle to give the caller control over the minimum time a job should take -
const throttle = (p, ms) =>
Promise.all([ p, sleep(ms) ]).then(([ value, _ ]) => value)
We can add a throttle in myEffect. Now if myJob runs very quickly, at least 5 seconds will pass before the next job is run -
async function myEffect(x) {
const r = await pool.run(_ => throttle(myJob(x), 5000))
const s = document.createTextNode(`${r}\n`)
document.body.appendChild(s)
return r
}

In general, it should be better to apply #Mulan answer.
But if you also stuck into cadesplugin.* generator functions and don't really care about heavyweight external libraries, this answer may also be helpful.
(If you are worried about heavyweight external libraries, you may still mix this answer with #Mulan's one)
Async task running can simply be solved using Promise.map function from bluebird library and double-usage of cadesplugin.async_spawn function.
The code will look like the following:
export function signData(dataItem) {
cadesplugin.async_spawn(async function* (args) {
// some extra logic before all of the tasks
await Promise.map(dataItem.identifiers,
(id) => cadesplugin.async_spawn(async function* (args) {
// ...
let oDocumentNameAttr = yield cadesplugin.CreateObjectAsync("CADESCOM.CPAttribute");
yield oDocumentNameAttr.propset_Value("Document Name");
// ...
// this function mutates some external data and making API calls
}),
{
concurrency: 5 //Parallel tasks count
});
// some extra logic after all tasks were finished
}, firstArg, secondArg);
}
The magic comes from async_spawn function which is defined as:
function async_spawn(generatorFunction) {
async function continuer(verb, arg) {
let result;
try {
result = await generator[verb](arg);
} catch (err) {
return Promise.reject(err);
}
if (result.done) {
return result.value;
} else {
return Promise.resolve(result.value).then(onFulfilled, onRejected);
}
}
let generator = generatorFunction(Array.prototype.slice.call(arguments, 1));
let onFulfilled = continuer.bind(continuer, "next");
let onRejected = continuer.bind(continuer, "throw");
return onFulfilled();
}
It can suspend the execution of internal generator functions on yield expressions without suspending the whole generator function.

How to test calling to an async not-awaited function

I want to test the execution of both fuctionUT and an inner async unwaited function externalCall passed by injection. The following code is simple working example of my functions and their usage:
const sleep = async (ms) => new Promise( (accept) => setTimeout(() => accept(), ms) )
const callToExternaService = async () => sleep(1000)
const backgroundJob = async (externalCall) => {
await sleep(500) // Simulate in app work
await externalCall() // Simulate external call
console.log('bk job done')
return 'job done'
}
const appDeps = {
externalService: callToExternaService
}
const functionUT = async (deps) => {
await sleep(30) // Simulate func work
// await backgroundJob(deps.externalService) // This make test work but slow down functionUT execution
backgroundJob(deps.externalService) // I don't want to wait for performance reason
.then( () => console.log('bk job ok') )
.catch( () => console.log('bk job error') )
return 'done'
}
functionUT( appDeps )
.then( (result) => console.log(result) )
.catch( err => console.log(err) )
module.exports = {
functionUT
}
Here there is a simple jest test case that fail but just for timing reasons:
const { functionUT } = require('./index')
describe('test', () => {
it('should pass', async () => {
const externaServiceMock = jest.fn()
const fakeDeps = {
externalService: externaServiceMock
}
const result = await functionUT(fakeDeps)
expect(result).toBe('done')
expect(externaServiceMock).toBeCalledTimes(1) //Here fail but just for timing reasons
})
})
What is the correct way to test the calling of externaServiceMock (make the test pass) without slowdown the performance of the functionUT ?
I have already found similar requests, but they threat only a simplified version of the problem.
how to test an embedded async call

You can't test for the callToExternaService to be called "somewhen later" indeed.
You can however mock backgroundJob and test that is was called with the expected arguments (before functionUT completes), as well as unit test backgroundJob on its own.

If a promise exists but cannot be reached in a place that relies on its settlement, this is a potential design problem. A module that does asynchronous side effects on imports is another problem. Both concerns affect testability, also they can affect the application if the way it works changes.
Considering there's a promise, you have an option to chain or not chain it in a specific place. This doesn't mean it should be thrown away. In this specific case it can be possibly returned from a function that doesn't chain it.
A common way to do this is to preserve a promise at every point in case it's needed later, at least for testing purposes, but probably for clean shutdown, extending, etc.
const functionUT = async (deps) => {
await sleep(30) // Simulate func work
return {
status: 'done',
backgroundJob: backgroundJob(deps.externalService)...
};
}
const initialization = functionUT( appDeps )...
module.exports = {
functionUT,
initialization
}
In this form it's supposed to be tested like:
beforeAll(async () => {
let result = await initialization;
await result.backgroundJob;
});
...
let result = await functionUT(fakeDeps);
expect(result.status).toBe('done')
await result.backgroundJob;
expect(externaServiceMock).toBeCalledTimes(1);
Not waiting for initialization can result in open handler if test suite is short enough and cause a reasonable warning from Jest.
The test can be made faster by using Jest fake timers in right places together with flush-promises.
functionUT( appDeps ) call can be extracted from the module to cause a side effect only in the place where it's needed, e.g. in entry point. This way it won't interfere with the rest of tests that use this module. Also at least some functions can be extracted to their own modules to be mockable and improve testability (backgroundJob, as another answer suggests) because they cannot be mocked separately when they are declared in the same module the way they are.

Better way to deal with Promise flow when functions need to access outer scope variable

I'm creating a Node.js module to interact with my API, and I use the superagent module to do the requests. How it works:
module.exports = data => {
return getUploadUrl()
.then(uploadFiles)
.then(initializeSwam)
function getUploadUrl() {
const request = superagent.get(....)
return request
}
function uploadFiles(responseFromGetUploadUrl) {
const request = superagent.post(responseFromGetUploadUrl.body.url)
// attach files that are in data.files
return request
}
function initializeSwam(responseFromUploadFilesRequest) {
// Same thing here. I need access data and repsonseFromUploadFilesRequest.body
}
}
I feel like I'm doing something wrong like that, but I can't think in a better way to achieve the same result.

Two simple ways:
write your function to take all parameters it needs
const doStuff = data =>
getUploadUrl()
.then(uploadFiles)
.then(initializeSwam)
might become
const doStuff = data =>
getUploadUrl()
.then(parseResponseUrl) // (response) => response.body.url
.then(url => uploadFiles(data, url))
.then(parseResponseUrl) // same function as above
.then(url => initializeSwam(data, url))
That should work just fine (or fine-ish, depending on what hand-waving you're doing in those functions).
partially apply your functions
const uploadFiles = (data) => (url) => {
return doOtherStuff(url, data);
};
// same deal with the other
const doStuff = data =>
getUploadUrl()
.then(parseResponseUrl)
.then(uploadFiles(data)) // returns (url) => { ... }
.then(parseResponseUrl)
.then(initializeSwam(data));
A mix of all of these techniques (when and where sensible) should be more than sufficient to solve a lot of your needs.

The way you have your code structured in the above snippet results in the getUploadUrl(), uploadFiles(), and initializeSwam() functions not being declared until the final .then(initializeSwam) call is made. What you have in this final .then() block is three function declarations, which simply register the functions in the namespace in which they are declared. A declaration doesn't fire-off a function.
I believe what you want is something like:
async function getUploadUrl() { <-- notice the flow control for Promises
const request = await superagent.get(....);
return request;
}
async function uploadFiles(responseFromGetUploadUrl) {
const request = await superagent.post(responseFromGetUploadUrl.body.url)
// attach files that are in data.files
return request;
}
async function initializeSwam(responseFromUploadFilesRequest) {
// Same thing here. I need access data and
repsonseFromUploadFilesRequest.body
const request = await ...;
}
module.exports = data => {
return getUploadUrl(data) <-- I'm guessing you wanted to pass this here
.then(uploadFiles)
.then(initializeSwam);
}
This approach uses ES6 (or ES2015)'s async/await feature; you can alternatively achieve the same flow control using the bluebird Promise library's coroutines paired with generator functions.

Using chrome.tabs.executeScript to execute an async function

I have a function I want to execute in the page using chrome.tabs.executeScript, running from a browser action popup. The permissions are set up correctly and it works fine with a synchronous callback:
chrome.tabs.executeScript(
tab.id,
{ code: `(function() {
// Do lots of things
return true;
})()` },
r => console.log(r[0])); // Logs true
The problem is that the function I want to call goes through several callbacks, so I want to use async and await:
chrome.tabs.executeScript(
tab.id,
{ code: `(async function() {
// Do lots of things with await
return true;
})()` },
async r => {
console.log(r); // Logs array with single value [Object]
console.log(await r[0]); // Logs empty Object {}
});
The problem is that the callback result r. It should be an array of script results, so I expect r[0] to be a promise that resolves when the script finishes.
Promise syntax (using .then()) doesn't work either.
If I execute the exact same function in the page it returns a promise as expected and can be awaited.
Any idea what I'm doing wrong and is there any way around it?

The problem is that events and native objects are not directly available between the page and the extension. Essentially you get a serialised copy, something like you will if you do JSON.parse(JSON.stringify(obj)).
This means some native objects (for instance new Error or new Promise) will be emptied (become {}), events are lost and no implementation of promise can work across the boundary.
The solution is to use chrome.runtime.sendMessage to return the message in the script, and chrome.runtime.onMessage.addListener in popup.js to listen for it:
chrome.tabs.executeScript(
tab.id,
{ code: `(async function() {
// Do lots of things with await
let result = true;
chrome.runtime.sendMessage(result, function (response) {
console.log(response); // Logs 'true'
});
})()` },
async emptyPromise => {
// Create a promise that resolves when chrome.runtime.onMessage fires
const message = new Promise(resolve => {
const listener = request => {
chrome.runtime.onMessage.removeListener(listener);
resolve(request);
};
chrome.runtime.onMessage.addListener(listener);
});
const result = await message;
console.log(result); // Logs true
});
I've extended this into a function chrome.tabs.executeAsyncFunction (as part of chrome-extension-async, which 'promisifies' the whole API):
function setupDetails(action, id) {
// Wrap the async function in an await and a runtime.sendMessage with the result
// This should always call runtime.sendMessage, even if an error is thrown
const wrapAsyncSendMessage = action =>
`(async function () {
const result = { asyncFuncID: '${id}' };
try {
result.content = await (${action})();
}
catch(x) {
// Make an explicit copy of the Error properties
result.error = {
message: x.message,
arguments: x.arguments,
type: x.type,
name: x.name,
stack: x.stack
};
}
finally {
// Always call sendMessage, as without it this might loop forever
chrome.runtime.sendMessage(result);
}
})()`;
// Apply this wrapper to the code passed
let execArgs = {};
if (typeof action === 'function' || typeof action === 'string')
// Passed a function or string, wrap it directly
execArgs.code = wrapAsyncSendMessage(action);
else if (action.code) {
// Passed details object https://developer.chrome.com/extensions/tabs#method-executeScript
execArgs = action;
execArgs.code = wrapAsyncSendMessage(action.code);
}
else if (action.file)
throw new Error(`Cannot execute ${action.file}. File based execute scripts are not supported.`);
else
throw new Error(`Cannot execute ${JSON.stringify(action)}, it must be a function, string, or have a code property.`);
return execArgs;
}
function promisifyRuntimeMessage(id) {
// We don't have a reject because the finally in the script wrapper should ensure this always gets called.
return new Promise(resolve => {
const listener = request => {
// Check that the message sent is intended for this listener
if (request && request.asyncFuncID === id) {
// Remove this listener
chrome.runtime.onMessage.removeListener(listener);
resolve(request);
}
// Return false as we don't want to keep this channel open https://developer.chrome.com/extensions/runtime#event-onMessage
return false;
};
chrome.runtime.onMessage.addListener(listener);
});
}
chrome.tabs.executeAsyncFunction = async function (tab, action) {
// Generate a random 4-char key to avoid clashes if called multiple times
const id = Math.floor((1 + Math.random()) * 0x10000).toString(16).substring(1);
const details = setupDetails(action, id);
const message = promisifyRuntimeMessage(id);
// This will return a serialised promise, which will be broken
await chrome.tabs.executeScript(tab, details);
// Wait until we have the result message
const { content, error } = await message;
if (error)
throw new Error(`Error thrown in execution script: ${error.message}.
Stack: ${error.stack}`)
return content;
}
This executeAsyncFunction can then be called like this:
const result = await chrome.tabs.executeAsyncFunction(
tab.id,
// Async function to execute in the page
async function() {
// Do lots of things with await
return true;
});
This wraps the chrome.tabs.executeScript and chrome.runtime.onMessage.addListener, and wraps the script in a try-finally before calling chrome.runtime.sendMessage to resolve the promise.

Passing promises from page to content script doesn't work, the solution is to use chrome.runtime.sendMessage and to send only simple data between two worlds eg.:
function doSomethingOnPage(data) {
fetch(data.url).then(...).then(result => chrome.runtime.sendMessage(result));
}
let data = JSON.stringify(someHash);
chrome.tabs.executeScript(tab.id, { code: `(${doSomethingOnPage})(${data})` }, () => {
new Promise(resolve => {
chrome.runtime.onMessage.addListener(function listener(result) {
chrome.runtime.onMessage.removeListener(listener);
resolve(result);
});
}).then(result => {
// we have received result here.
// note: async/await are possible but not mandatory for this to work
logger.error(result);
}
});

For anyone who is reading this but using the new manifest version 3 (MV3), note that this should now be supported.
chrome.tabs.executeScript has been replaced by chrome.scripting.executeScript, and the docs explicitly state that "If the [injected] script evaluates to a promise, the browser will wait for the promise to settle and return the resulting value."