Puppeteer Script Works Local But Not On EC2 AWS - javascript

I am running the following script on EC2 AWS with Amazon Linux AMI
const puppeteer = require('puppeteer');
url_ = 'https://www.barchart.com/futures/quotes/ES*0/interactive-chart/fullscreen';
func()
async function func() {
console.log(0)
const browser = await puppeteer.launch();
console.log(1)
const page = await browser.newPage();
console.log(2)
await page.goto(url_);
console.log(page)
return page
}
since today, with no changes on or new installs on EC2, the script stopped working while it was working until yesterday.
The same script on Local Machine still works.
Instead on EC2 gives the following error:
(node:12636) UnhandledPromiseRejectionWarning: Error: Navigation failed because browser has disconnected!
at CDPSession.LifecycleWatcher._eventListeners.helper.addEventListener (/home/ec2-user/hd/node_modules/puppeteer/lib/LifecycleWatcher.js:47:107)
at CDPSession.emit (events.js:189:13)
at CDPSession._onClosed (/home/ec2-user/hd/node_modules/puppeteer/lib/Connection.js:215:10)
at Connection._onClose (/home/ec2-user/hd/node_modules/puppeteer/lib/Connection.js:138:15)
at WebSocketTransport._ws.addEventListener.event (/home/ec2-user/hd/node_modules/puppeteer/lib/WebSocketTransport.js:45:22)
at WebSocket.onClose (/home/ec2-user/hd/node_modules/ws/lib/event-target.js:124:16)
at WebSocket.emit (events.js:189:13)
at WebSocket.emitClose (/home/ec2-user/hd/node_modules/ws/lib/websocket.js:191:10)
at Socket.socketOnClose (/home/ec2-user/hd/node_modules/ws/lib/websocket.js:850:15)
at Socket.emit (events.js:189:13)
-- ASYNC --
at Frame. (/home/ec2-user/hd/node_modules/puppeteer/lib/helper.js:110:27)
at Page.goto (/home/ec2-user/hd/node_modules/puppeteer/lib/Page.js:656:49)
at Page. (/home/ec2-user/hd/node_modules/puppeteer/lib/helper.js:111:23)
at func (/home/ec2-user/hd/scrape_bk.js:19:13)
at process._tickCallback (internal/process/next_tick.js:68:7)
(node:12636) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:12636) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
if I give instead this URL (http://www.google.com) I get this error on EC2 (while it still works on local machine):
(node:12938) UnhandledPromiseRejectionWarning: Error: Page crashed!
at Page._onTargetCrashed (/home/ec2-user/hd/node_modules/puppeteer/lib/Page.js:185:24)
at CDPSession.Page.client.on.event (/home/ec2-user/hd/node_modules/puppeteer/lib/Page.js:140:56)
at CDPSession.emit (events.js:189:13)
at CDPSession._onMessage (/home/ec2-user/hd/node_modules/puppeteer/lib/Connection.js:200:12)
at Connection._onMessage (/home/ec2-user/hd/node_modules/puppeteer/lib/Connection.js:112:17)
at WebSocketTransport._ws.addEventListener.event (/home/ec2-user/hd/node_modules/puppeteer/lib/WebSocketTransport.js:41:24)
at WebSocket.onMessage (/home/ec2-user/hd/node_modules/ws/lib/event-target.js:120:16)
at WebSocket.emit (events.js:189:13)
at Receiver.receiverOnMessage (/home/ec2-user/hd/node_modules/ws/lib/websocket.js:789:20)
at Receiver.emit (events.js:189:13)
(node:12938) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:12938) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
(node:12938) UnhandledPromiseRejectionWarning: Error: Navigation failed because browser has disconnected!
at CDPSession.LifecycleWatcher._eventListeners.helper.addEventListener (/home/ec2-user/hd/node_modules/puppeteer/lib/LifecycleWatcher.js:47:107)
at CDPSession.emit (events.js:189:13)
at CDPSession._onClosed (/home/ec2-user/hd/node_modules/puppeteer/lib/Connection.js:215:10)
at Connection._onClose (/home/ec2-user/hd/node_modules/puppeteer/lib/Connection.js:138:15)
at WebSocketTransport._ws.addEventListener.event (/home/ec2-user/hd/node_modules/puppeteer/lib/WebSocketTransport.js:45:22)
at WebSocket.onClose (/home/ec2-user/hd/node_modules/ws/lib/event-target.js:124:16)
at WebSocket.emit (events.js:189:13)
at WebSocket.emitClose (/home/ec2-user/hd/node_modules/ws/lib/websocket.js:191:10)
at Socket.socketOnClose (/home/ec2-user/hd/node_modules/ws/lib/websocket.js:850:15)
at Socket.emit (events.js:189:13)
-- ASYNC --
at Frame. (/home/ec2-user/hd/node_modules/puppeteer/lib/helper.js:110:27)
at Page.goto (/home/ec2-user/hd/node_modules/puppeteer/lib/Page.js:656:49)
at Page. (/home/ec2-user/hd/node_modules/puppeteer/lib/helper.js:111:23)
at func (/home/ec2-user/hd/scrape_bk.js:13:13)
at process._tickCallback (internal/process/next_tick.js:68:7)
(node:12938) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
I have no idea how to identify the source of the problem since it came out suddenly with no changes to machine or code and on my local machine it works fine.

It seems puppeteer is able to start the browser but it then randomly crashes. This should not be happening on a normal machine and might be happening due to resource constraints.
You should check if your system is running correctly. In particular, you might want to check:
Memory usage
CPU usage
Disk usage
Too few memory or not enough CPU power might lead to random crashes as well as not enough hard disk space.

Related

Unable to get local issuer certificate in node js

I am new to the node js.I am getting this error while running my node js code in local. Please help me in this.
(node:7532) UnhandledPromiseRejectionWarning: Error: unable to get local issuer certificate
at TLSSocket.onConnectSecure (_tls_wrap.js:1048:34)
at TLSSocket.emit (events.js:182:13)
at TLSSocket._finishInit (_tls_wrap.js:628:8)
(node:7532) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:7532) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
I tried some of the available solution on the internet, but so far any of them are working.
This are the things I have tried.
npm set strict-ssl=false
npm config set strict-ssl false
export NODE_TLS_REJECT_UNAUTHORIZED=0

I get "TypeError: Cannot read property 'tag' of null" but only in the packaged version of my discord.js bot

I am working on a discord bot in discord.js and everything has been working just fine. But I want to compile it into a .exe file for ease of use, etc. I can run the bot fine just using node.js but when I run the .exe I get the following error message:
(node:3656) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'tag' of null
at Client.<anonymous> (C:\snapshot\src\index.js:19:42)
at Client.emit (events.js:315:20)
at WebSocketManager.triggerClientReady (C:\snapshot\src\node_modules\discord.js\src\client\websocket\WebSocketManager.js:431:17)
at WebSocketManager.checkShardsReady (C:\snapshot\src\node_modules\discord.js\src\client\websocket\WebSocketManager.js:415:10)
at WebSocketShard.<anonymous> (C:\snapshot\src\node_modules\discord.js\src\client\websocket\WebSocketManager.js:197:14)
at WebSocketShard.emit (events.js:315:20)
at WebSocketShard.checkReady (C:\snapshot\src\node_modules\discord.js\src\client\websocket\WebSocketShard.js:475:12)
at WebSocketShard.onPacket (C:\snapshot\src\node_modules\discord.js\src\client\websocket\WebSocketShard.js:447:16)
at WebSocketShard.onMessage (C:\snapshot\src\node_modules\discord.js\src\client\websocket\WebSocketShard.js:301:10)
at WebSocket.onMessage (C:\snapshot\src\node_modules\ws\lib\event-target.js:132:16)
(Use `jamesinasbot --trace-warnings ...` to show where the warning was created)
(node:3656) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:3656) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
This is the line with the issue:
console.log(`Logged in as ${client.user.tag}.`);
I am not sure why this happens. I have made sure all the dependencies are included, it is in the same directory as index.js, and since it is only saying it can't read the property tag of null it seems to be an issue with variables or something not dependencies. I have tried using --trace-warnings but it doesn't show anything different.
Any help is appreciated!

AWS Amplify error: (node:68674) UnhandledPromiseRejectionWarning: NoSuchKey: The specified key does not exist

I am applying Amplify in Vue.js,
so the issue here is after I've done command successfully :
amplify add api
then I am trying to do command:
amplify push
Then below error always there:
SKY-MacBook-Pro:my-project sky$ amplify push ⠇ Fetching updates to
backend environment: mpvue from the cloud.(node:68674)
UnhandledPromiseRejectionWarning: NoSuchKey: The specified key does
not exist.
at Request.extractError (/usr/local/lib/node_modules/#aws-amplify/cli/node_modules/aws-sdk/lib/services/s3.js:816:35)
at Request.callListeners (/usr/local/lib/node_modules/#aws-amplify/cli/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/usr/local/lib/node_modules/#aws-amplify/cli/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/usr/local/lib/node_modules/#aws-amplify/cli/node_modules/aws-sdk/lib/request.js:683:14)
at Request.transition (/usr/local/lib/node_modules/#aws-amplify/cli/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/usr/local/lib/node_modules/#aws-amplify/cli/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /usr/local/lib/node_modules/#aws-amplify/cli/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request. (/usr/local/lib/node_modules/#aws-amplify/cli/node_modules/aws-sdk/lib/request.js:38:9)
at Request. (/usr/local/lib/node_modules/#aws-amplify/cli/node_modules/aws-sdk/lib/request.js:685:12)
at Request.callListeners (/usr/local/lib/node_modules/#aws-amplify/cli/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
(node:68674) UnhandledPromiseRejectionWarning: Unhandled promise
rejection. This error originated either by throwing inside of an async
function without a catch block, or by rejecting a promise which was
not handled with .catch(). (rejection id: 1) (node:68674) [DEP0018]
DeprecationWarning: Unhandled promise rejections are deprecated. In
the future, promise rejections that are not handled will terminate the
Node.js process with a non-zero exit code. ⠹ Fetching updates to
backend environment: mpvue from the cloud. SKY-MacBook-Pro:my-project
sky$
Here is the docs for your reference:
https://aws-amplify.github.io/docs/js/start?platform=vue

Puppeteer cluster example throwing Error: Unable to get browser page

I'm using this example in the puppeteer-cluster docs
I am running this on node v10.15.3 I have attempted passing the headless property and slowMo to puppeteer options.
I would expect the code to log out and create a Screenshot of the page however, what happens is several chrome instances boot up which do not load any of the pages then the console hangs and errors with:
(node:30826) UnhandledPromiseRejectionWarning: Error: Unable to get browser page
at Worker.<anonymous> (/Users/Starlord/Code/oppose/node_modules/puppeteer-cluster/dist/Worker.js:43:31)
at Generator.next (<anonymous>)
at fulfilled (/Users/Starlord/Code/oppose/node_modules/puppeteer-cluster/dist/Worker.js:4:58)
at process._tickCallback (internal/process/next_tick.js:68:7)
(node:30826) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 21)
(node:30826) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
(node:30826) UnhandledPromiseRejectionWarning: Error: Unable to get browser page
at Worker.<anonymous> (/Users/Starlord/Code/oppose/node_modules/puppeteer-cluster/dist/Worker.js:43:31)
at Generator.next (<anonymous>)
at fulfilled (/Users/Starlord/Code/oppose/node_modules/puppeteer-cluster/dist/Worker.js:4:58)
at process._tickCallback (internal/process/next_tick.js:68:7)
(node:30826) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 22)
^Cinternal/process/per_thread.js:198
throw errnoException(err, 'kill');
^
Error: kill ESRCH
at process.kill (internal/process/per_thread.js:198:13)
at process.killChrome (/Users/Starlord/Code/oppose/node_modules/puppeteer/lib/Launcher.js:110:17)
at process.emit (events.js:189:13)
This error happens when the library is unable to create a new browser page. The error is thrown here after several tries of opening a new context/page/browser (depending on your settings).
Debugging log
To get more information on what causes the error, you can check the debugging log. This starts the application in debugging mode and will log more information. To enter debugging mode you need to start the application with the environment variable DEBUG='puppeteer-cluster:*':
# Linux
DEBUG='puppeteer-cluster:*' node application.js
# Windows Powershell
$env:DEBUG='puppeteer-cluster:*';node application.js
Problem
In your case the debug logs shows (copied from your comment above):
puppeteer-cluster: Worker Error getting browser page (try: 9), message: this.browser.createIncognitoBrowserContext is not a function +660ms
This means, there is no createIncognitoBrowserContext that the library can use to create a new context. As you already confirmed, this is the case, because you are using an old puppeteer installation. To use the setting { concurrency: Cluster.CONCURRENCY_CONTEXT }, you have to be using at least version 1.5.0 as that was when contexts were introduced.
Fix
To fix the problem, update your puppeteer installation:
npm install puppeteer#latest
The problem was that I was using puppeteer-pool before puppeteer-cluster which uses an older version of puppeteer.
I removed puppeteer-pool and installed puppeteer and it works as expected

Selenium with NodeJS - the driver object loses context when returning from TinyMCE iframe

First, I login on a separate page to have access to the page with the form.
After logging in correctly and after switching to the page with the form, I have access to all its elements and I can freely edit them. One of the form element is the TinyMCE iframe.
After entering the iframe and modifying it, I don't know how to return to the native form and click on submit...
Here is my code (my.js):
const loginUrl = 'https://www.example.com/login';
const formUrl = 'https://www.example.com/form';
const {Builder, By, until} = require('selenium-webdriver');
const driver_ch = new Builder().forBrowser('chrome').build();
driver_ch.manage().window().maximize();
driver_ch.get(loginUrl);
const loginLink = driver_ch.wait(until.elementLocated(By.css('#login-box')), 20000);
loginLink.click().then(function() {
driver_ch.wait(until.elementLocated(By.css('#login'))).sendKeys('user_name');
driver_ch.wait(until.elementLocated(By.css('#password'))).sendKeys('secret_password');
driver_ch.wait(until.elementLocated(By.css('#submit-login'))).click();
}).then(function() {
driver_ch.wait(until.elementLocated(By.css('.form-action'))).then(function() {
driver_ch.get(formUrl);
driver_ch.switchTo().frame(driver_ch.findElement(By.id("edit-body-und-0-value_ifr")));
driver_ch.wait(until.elementLocated(By.id('tinymce'))).sendKeys('Glory! Glory! Hallelujah!');
Until now everything work, but this not:
driver_ch.switchTo().defaultContent();
and this also not:
driver_ch.switchTo().parentFrame();
so this could not be done...
driver_ch.findElement(By.css('#edit-submit')).click();
});
});
Error messege after runing my.js script:
node my.js
(node:3955) UnhandledPromiseRejectionWarning: StaleElementReferenceError: stale element reference: element is not attached to the page document
(Session info: chrome=74.0.3729.169)
(Driver info: chromedriver=2.41.578700 (2f1ed5f9343c13f73144538f15c00b370eda6706),platform=Linux 4.9.0-9-amd64 x86_64)
at Object.checkLegacyResponse (/home/user1/test/node_modules/selenium-webdriver/lib/error.js:585:15)
at parseHttpResponse (/home/user1/test/node_modules/selenium-webdriver/lib/http.js:554:13)
at Executor.execute (/home/user1/test/node_modules/selenium-webdriver/lib/http.js:489:26)
at process._tickCallback (internal/process/next_tick.js:68:7)
(node:3955) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:3955) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Changing the driver from Chrome to FireFox did not solve the problem.
Edited:
Thank you #DebanjanB for your suggestions in your answer.
I have updated the Chrome browser and Chrome driver as you suggested, but unfortunately it still doesn't work...
I get this error message now:
node my.js
(node:24587) UnhandledPromiseRejectionWarning: StaleElementReferenceError: stale element reference: element is not attached to the page document
(Session info: chrome=78.0.3904.70)
at Object.throwDecodedError (/home/user1/test/node_modules/selenium-webdriver/lib/error.js:550:15)
at parseHttpResponse (/home/user1/test/node_modules/selenium-webdriver/lib/http.js:563:13)
at Executor.execute (/home/user1/test/node_modules/selenium-webdriver/lib/http.js:489:26)
at process._tickCallback (internal/process/next_tick.js:68:7)
(node:24587) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promis
e which was not handled with .catch(). (rejection id: 1)
(node:24587) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zer
o exit code.
This error message...
node my.js
(node:3955) UnhandledPromiseRejectionWarning: StaleElementReferenceError: stale element reference: element is not attached to the page document
(Session info: chrome=74.0.3729.169)
(Driver info: chromedriver=2.41.578700 (2f1ed5f9343c13f73144538f15c00b370eda6706),platform=Linux 4.9.0-9-amd64 x86_64)
at Object.checkLegacyResponse (/home/user1/test/node_modules/selenium-webdriver/lib/error.js:585:15)
at parseHttpResponse (/home/user1/test/node_modules/selenium-webdriver/lib/http.js:554:13)
at Executor.execute (/home/user1/test/node_modules/selenium-webdriver/lib/http.js:489:26)
at process._tickCallback (internal/process/next_tick.js:68:7)
(node:3955) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:3955) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
...implies that the ChromeDriver was unable to interact with the desired element i.e. the defaultContent / parentFrame.
I don't see any major issue in your code trials. However your main issue seems to be incompatibility between the version of the binaries you are using as follows:
You are using chromedriver=2.41
Release Notes of chromedriver=2.41 clearly mentions the following :
Supports Chrome v67-69
You are using chrome= chrome=74.0
Release Notes of ChromeDriver v74.0.3729.6 clearly mentions the following :
Supports Chrome v74
So there is a clear mismatch between the ChromeDriver v2.41 and the Chrome Browser v74.0
Solution
Ensure that:
ChromeDriver is updated to current ChromeDriver v78.0 level.
Chrome is updated to current Chrome Version 78.0 level. (as per ChromeDriver v78.0 release notes)
Clean your Project Workspace through your IDE and Rebuild your project with required dependencies only.
If your base Web Client version is too old, then uninstall it and install a recent GA and released version of Web Client.
Take a System Reboot.
Execute your #Test.

Categories