I set up a loop inside of an async function to iterate over a list of URLs. The data is coming from an .xls file converted to .json. My goal is to capture a screenshot of each URL in the array but I keep getting UnhandledPromiseRejectionWarning. Any ideas how how i might achieve this? Any help is appreciated!
The Code:
const puppeteer = require("puppeteer");
const excel = require("./excel");
const data = excel.data;
async function run(arr) {
for (let i = 0; i < data.length; i++) {
const url = data[i]["Seller"];
const sku = data[i]["Seller Name"];
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.setViewport({
width: 1000,
height: 840,
deviceScaleFactor: 1
});
await page.goto(url, { waitUntil: "load" });
await page.screenshot({ path: `screenshots/${sku}.jpg` });
await browser.close();
}
}
run(data)
full error with try ... catch
(node:24804) UnhandledPromiseRejectionWarning: ReferenceError: err is not defined
at run (C:\Users\ray\OneDrive\Desktop\nodeScrappingProject\app\index.js:24:19)
(node:24804) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which
was not handled with .catch(). (rejection id: 1)
(node:24804) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Full error message without try .. catch
(node:34456) UnhandledPromiseRejectionWarning: Error: ENOENT: no such file or directory, open 'C:\Users\ray\OneDrive\Desktop\nodeScrappingProject\app\screenshots\fileName.jpg'
-- ASYNC --
at Page.<anonymous> (C:\Users\ray\OneDrive\Desktop\nodeScrappingProject\node_modules\puppeteer\lib\helper.js:111:15)
at run (C:\Users\ray\OneDrive\Desktop\nodeScrappingProject\app\index.js:20:16)
at processTicksAndRejections (internal/process/task_queues.js:85:5)
(node:34456) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which
was not handled with .catch(). (rejection id: 1)
(node:34456) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Your code seems correct to me. Did you try looking at what is coming out in data Seller and SellerName? I am guessing that you have issues in data. As you are not asserting anything you need not close the browser each time. Checkout the following snippet.
const puppeteer = require('puppeteer');
(async () => {
const data = [{"sellerUrl" : "https://www.amazon.com/",
"sellerName" : "amazon"
},
{"sellerUrl" : "https://www.ebay.com/",
"sellerName" : "ebay"
},
];
const browser = await puppeteer.launch();
const page = await browser.newPage();
for(const x of data){
await page.goto(x.sellerUrl, { waitUntill: 'networkidle2'});
await page.screenshot({path: `${x.sellerName}.jpg`});
}
await browser.close();
})();
Related
I am trying to use clipboard in order to copy 'longlat' to the clipboard by passing it through the function 'copyData()'. However, it keeps giving me errors.
This is the error (1. Clipboardy.writeSync not a function and 2.UnhandledPromiseRejectionWarning) :
(node:16596) UnhandledPromiseRejectionWarning: TypeError: clipboardy.writeSync is not a function
at copyData (C:\Users\Nicky\index.js:46:14)
at C:\Users\Nicky\index.js:32:13
at processTicksAndRejections (internal/process/task_queues.js:97:5)
(node:16596) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag --unhandled-rejections=strict (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:16596) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
This is my code
const clipboardy = (...args) => import('clipboardy').then(({default: clipboardy}) => clipboardy(...args));
const puppeteer = require("puppeteer");
const url = "https://www.google.com/";
async function StartScraping() {
await puppeteer
.launch({
headless: false,
})
.then(async (browser) => {
const page = await browser.newPage();
await page.setViewport({
width: 2000,
height: 800,
});
page.on("response", async (response) => {
if (response.url().includes("General")) {
const location = await response.text()
let intlong = location.indexOf("Example:")
const longlat = location.substring(intlong, intlong + 46)
console.log(longlat);
copyData(longlat)
}
});
await page.goto(url, {
waitUntil: "load",
timeout: 0,
});
});
}
function copyData(data1) {
clipboardy.writeSync(data1)
}
StartScraping();
cy.window().its('navigator.clipboard')
.invoke('readText').then((data) => {
console.log(data);
})
Should do the trick. If you are using cypress on the browser
I am trying to log all network requests in a page. My goal is to later filter for particular requests and check their initiator. The code below runs fine if I comment line 12 const request_initiator = interceptedRequest.initiator();. But the initiator is something really need. According to puppeteer's docs I am calling it correctly.
Here is my code:
const puppeteer = require('puppeteer');
(async() => {
const results = [];
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.setRequestInterception(true);
page.on('request', interceptedRequest => {
const request_url = interceptedRequest.url();
const request_method = interceptedRequest.method();
const request_payload = interceptedRequest.postData();
const request_initiator = interceptedRequest.initiator();
results.push({
request_url,
request_method,
request_payload,
request_initiator
})
interceptedRequest.continue();
});
await page.goto('https://testurl.com');
await browser.close();
return results
})().then(function(result) {
console.log(result)
});
Here is the error I am getting:
UnhandledPromiseRejectionWarning: TypeError: interceptedRequest.initiator is not a function
at /Users/test/Desktop/code/tagv2/tcv2/index.js:12:54
at /Users/test/Desktop/code/tagv2/tcv2/node_modules/peteer/lib/cjs/vendor/mitt/src/index.js:51:62
at Array.map (<anonymous>)
at Object.emit (/Users/test/Desktop/code/tagv2/tcv2/node_modules/puppeteer/lib/cjs/vendor/mitt/src/index.js:51:43)
at Page.emit (/Users/test/Desktop/code/tagv2/tcv2/node_modules/puppeteer/lib/cjs/puppeteer/common/EventEmitter.js:72:22)
at /Users/test/Desktop/code/tagv2/tcv2/node_modules/puppeteer/lib/cjs/puppeteer/common/Page.js:143:100
at /Users/test/Desktop/code/tagv2/tcv2/node_modules/puppeteer/lib/cjs/vendor/mitt/src/index.js:51:62
at Array.map (<anonymous>)
at Object.emit (/Users/test/Desktop/code/tagv2/tcv2/node_modules/puppeteer/lib/cjs/vendor/mitt/src/index.js:51:43)
at NetworkManager.emit (/Users/test/Desktop/code/tagv2/tcv2/node_modules/puppeteer/lib/cjs/puppeteer/common/EventEmitter.js:72:22)
(node:22974) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:22974) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
I'm running the following node.js code in BigQuery but I'm getting an unhandled promise error:
I'm not too familiar with node.js - do I have to add a .catch() in there at the end?
const BigQuery = require('#google-cloud/bigquery');
const bigquery = new BigQuery({
projectId: 'xxxxx',
keyFilename: 'key.json'
});
const query = `SELECT total_amount, pickup_datetime, trip_distance
FROM \`nyc-tlc.yellow.trips\`
ORDER BY total_amount DESC
LIMIT 1;`
bigquery.createQueryJob(query).then((data) => {
const job = data[0];
return job.getQueryResults({timeoutMs: 10000});
}).then((data) => {
const rows = data[0];
console.log(rows[0]);
});
Error:
(node:1035) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either
by throwing inside of an async function without a catch block, or by rejecting a promise which was not
handled with .catch(). (rejection id: 1)
(node:1035) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, p
romise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Thanks
I'm not familiar with #google-cloud/bigquery anyway you should do :
.then((data) => {
const rows = data[0];
console.log(rows[0]);
}).catch(e=>{
//handle exception
console.log(e)
})
Overview
I'm looking for a simpler way to handle clicking on links which open new pages (like target="_blank" anchor tags).
Here handle means:
get the new page object
wait for the new tab to load (with timeout)
Steps to reproduce
Tell us about your environment:
Puppeteer version: ^1.11.0
Platform / OS version: 64-bit, win 10 pro
URLs (if applicable): none
Node.js version: v10.15.0
I've looked at related issues:
https://github.com/GoogleChrome/puppeteer/issues/386
https://github.com/GoogleChrome/puppeteer/issues/3535
https://github.com/GoogleChrome/puppeteer/issues/978
and more
What steps will reproduce the problem?
I've included the code snippet below
I'm trying to:
Get the object for the new page when clicking on a link opens a new tab. (The links are dynamically generated, capturing href might not be the most elegant way)
Wait till the new page loads (with timeout). I'd like it if you can use page.waitForNavigation for consistency
close the tab and return the earlier tab to continue further operations
Please include code that reproduces the issue.
// as referenced here on #386 : https://github.com/GoogleChrome/puppeteer/issues/386#issuecomment-425109457
const getNewPageWhenLoaded = async () => {
return new Promise(x =>
global.browser.on('targetcreated', async target => {
if (target.type() === 'page') {
const newPage = await target.page();
const newPagePromise = new Promise(y =>
newPage.once('domcontentloaded', () => y(newPage))
);
const isPageLoaded = await newPage.evaluate(
() => document.readyState
);
return isPageLoaded.match('complete|interactive')
? x(newPage)
: x(newPagePromise);
}
})
);
};
const newPagePromise = getNewPageWhenLoaded();
await page.click('my-link'); // or just do await page.evaluate(() => window.open('https://www.example.com/'));
const newPage = await newPagePromise;
What is the expected result?
An easier and consistent way to handle new tabs
What happens instead?
The developer has to write what looks like plumbing (internal/ low level) commands.
Usage of waitForTarget might simplify this, but I've not been able to get the predicate to return the right types. Here's my non-functional code
private async getNewPageWhenLoaded() {
const newTarget = await this._browser.waitForTarget(async (target) => {
const newPage = await target.page();
await newPage.waitForNavigation(this._optionsNavigation);
// const newPagePromise = new Promise(() => newPage.once('load', () => x(newPage)));
return await newPage.evaluate("true");
});
return await newTarget.page();
}
// elsewhere in the code
const newPagePromise = this.getNewPageWhenLoaded();
await resultItem.element.click();
const newPage = <Page>await newPagePromise;
//I get the following error
DevTools listening on ws://127.0.0.1:31984/devtools/browser/bf86648d-d52d-42d8-a392-629bf96211d4
(node:5564) UnhandledPromiseRejectionWarning: Error: Navigation failed because browser has disconnected!
at CDPSession.LifecycleWatcher._eventListeners.helper.addEventListener (<path-to-my-project>\node_modules\puppeteer\lib\FrameManager.js:1181:107)
at CDPSession.emit (events.js:182:13)
at CDPSession._onClosed (<path-to-my-project>\node_modules\puppeteer\lib\Connection.js:231:10)
at Connection._onMessage (<path-to-my-project>\node_modules\puppeteer\lib\Connection.js:103:19)
at WebSocketTransport._ws.addEventListener.event (<path-to-my-project>\node_modules\puppeteer\lib\WebSocketTransport.js:41:24)
at WebSocket.onMessage (<path-to-my-project>\node_modules\ws\lib\event-target.js:120:16)
at WebSocket.emit (events.js:182:13)
at Receiver.receiverOnMessage (<path-to-my-project>\node_modules\ws\lib\websocket.js:741:20)
at Receiver.emit (events.js:182:13)
at Receiver.dataMessage (<path-to-my-project>\node_modules\ws\lib\receiver.js:417:14)
-- ASYNC --
at Frame.<anonymous> (<path-to-my-project>\node_modules\puppeteer\lib\helper.js:144:27)
at Page.waitForNavigation (<path-to-my-project>\node_modules\puppeteer\lib\Page.js:644:49)
at Page.<anonymous> (<path-to-my-project>\node_modules\puppeteer\lib\helper.js:145:23)
at newTarget._browser.waitForTarget (<path-to-my-project>\pageObjects\MyPage.js:104:27)
at process._tickCallback (internal/process/next_tick.js:68:7)
(node:5564) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:5564) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
(node:5564) UnhandledPromiseRejectionWarning: TimeoutError: Navigation Timeout Exceeded: 300000ms exceeded
at Promise.then (<path-to-my-project>\node_modules\puppeteer\lib\FrameManager.js:1276:21)
-- ASYNC --
at Frame.<anonymous> (<path-to-my-project>\node_modules\puppeteer\lib\helper.js:144:27)
at Page.waitForNavigation (<path-to-my-project>\node_modules\puppeteer\lib\Page.js:644:49)
at Page.<anonymous> (<path-to-my-project>\node_modules\puppeteer\lib\helper.js:145:23)
at newTarget._browser.waitForTarget (<path-to-my-project>\pageObjects\MyPage.js:104:27)
at process._tickCallback (internal/process/next_tick.js:68:7)
(node:5564) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
Note: issue I've created on github: https://github.com/GoogleChrome/puppeteer/issues/3718
first run the click function first of all and remove "global" inside the promise and declaring browser as constant outside the promise
const browser = await puppeteer.launch();
await page.click('my-link');
const getNewPageWhenLoaded = async () => {
return new Promise(x =>
browser.on('targetcreated', async target => {
if (target.type() === 'page') {
const newPage = await target.page();
const newPagePromise = new Promise(y =>
newPage.once('domcontentloaded', () => y(newPage))
);
const isPageLoaded = await newPage.evaluate(
() => document.readyState
);
return isPageLoaded.match('complete|interactive')
? x(newPage)
: x(newPagePromise);
}
})
);
};
const newPagePromise = getNewPageWhenLoaded();
const newPage = await newPagePromise;
in my async function i use Promise.all but for some reason its not defined or something here is the function
async function check_available_money() {
global_browser = await puppeteer.launch({headless: false, args: ['--no-sandbox', '--disable-setuid-sandbox']});
const page = await global_browser.newPage();
await page.setViewport({width: 1000, height: 1100});
var setting = {'username': 'aa', 'password': 'bb'};
try {
await page.goto('https://example.com/login', {timeout: 90000})
.catch(function (error) {
throw new Error(' TIMEOUT 1 ');
}
);
await page.$eval('#username', (el, setting) => el.value = setting.username, setting);
await page.$eval('#password', (el, setting) => el.value = setting.password, setting);
console.log(tab_id + ' -> SUMITING LOGIN FORM ');
await Promise.all(
page.$eval('form', form => form.submit()),
page.waitForNavigation()
)
console.log(tab_id + ' -> SUMITING LOGIN FORM DONE ! ');
}
catch (e) {
await page.close();
console.log(e);
}
}
i get error from this part
await Promise.all(
page.$eval('form', form => form.submit()),
page.waitForNavigation()
)
if i remove await Promise.all and just type
await page.$eval('form', form => form.submit());
await page.waitForNavigation();
it works ok
here is error stack
TypeError: undefined is not a function
at Function.all (<anonymous>)
at check_available_money (D:\wamp\www\withdraw\robot\server.js:115:23)
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:188:7)
(node:13184) UnhandledPromiseRejectionWarning: Error: Protocol error (Runtime.callFunctionOn): Target closed.
at Promise (D:\wamp\www\withdraw\robot\node_modules\puppeteer\lib\Connection.js:202:56)
at new Promise (<anonymous>)
at CDPSession.send (D:\wamp\www\withdraw\robot\node_modules\puppeteer\lib\Connection.js:201:12)
at ExecutionContext.evaluateHandle (D:\wamp\www\withdraw\robot\node_modules\puppeteer\lib\ExecutionContext.js:79:75)
at ExecutionContext.evaluate (D:\wamp\www\withdraw\robot\node_modules\puppeteer\lib\ExecutionContext.js:46:31)
at ElementHandle.$eval (D:\wamp\www\withdraw\robot\node_modules\puppeteer\lib\ElementHandle.js:293:50)
at <anonymous>
at process._tickCallback (internal/process/next_tick.js:188:7)
(node:13184) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 3)
(node:13184) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
(node:13184) UnhandledPromiseRejectionWarning: Error: Navigation Timeout Exceeded: 30000ms exceeded
at Promise.then (D:\wamp\www\withdraw\robot\node_modules\puppeteer\lib\NavigatorWatcher.js:73:21)
at <anonymous>
(node:13184) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 4)
Promise.all takes an iterable, not multiple arguments. It tried to iterate your first argument, but that didn't have a [Symbol.iterator] method - "undefined is not a function". Pass an array:
await Promise.all([
page.$eval('form', form => form.submit()),
page.waitForNavigation(),
])