We Keep Coding

couldn't grab input element from a website

so I'm trying to get an input element from Twitter but when I run it, it keeps giving me an error like this in the node terminal, as a result, a browser window made from this code will close itself because it doesn't find the right input selector. how do I grab the right kind of input?
Error: No element found for selector: input[name="text"]
const puppeteer = require('puppeteer');
(async () => {
const browser = await puppeteer.launch({
headless: false,
})
const page = await browser.newPage();
page.setViewport({
width: 1280,
height: 800,
isMobile: false
})
await page.goto("https://twitter.com/i/flow/login");
await page.type('input[name="text"]', 'username', {delay: 25})
})();
i tried different selectors including class attribute but still gets error

you need to waitForSelector to appear on the page before typing. That is why you got an error, it couldn't find the element.
await page.waitForSelector('input[name="text"]');
await page.type('input[name="text"]', 'username', {delay: 25})

Puppeteer Application Error: A client side exception has occurred

I am using Puppeteer with NEXT.JS, trying to take a screenshot. And it works fine on localhost but returns an image with this error in production:
Application error a client-side exception has occurred (see the browser console for more information!!
Taking a screenshot
export const createImages = async (urlArray) => {
try {
const browser = await puppeteer.launch({
headless: true,
args: [
"--no-sandbox",
"--disable-setuid-sandbox",
"--disable-dev-shm-usage",
],
slowMo: 250, // slow down by 250ms
})
const page = await browser.newPage()
for (let i = 0; i < urlArray.length; i++) {
if (urlArray[i].address === "") continue
await page.goto(urlArray[i].address, {
waitUntil: "load",
timeout: 30000,
})
const screenshotBase64 = await page.screenshot({
encoding: "base64",
})
const screenshot = Buffer.from(
await screenshotBase64.replace(/^data:image\/\w+;base64,/, ""),
"base64"
)
urlArray[i]["imgBase64"] = screenshot
}
await browser.close()
} catch (err) {
console.log(new Date(), "was not able to create images: ", err)
return err
}
return 1
}
When I open the url manually in production, the page loads fine! And I have tried encoding the image to Binary instead but still the same issue.. Any idea !?

At first I was listening only to the errors. But after I listened to all console messages using this command:
page.on('console', msg => console.log('PAGE LOG:', msg.text()))
I was able to see this error:
'THREE.WebGLRenderer: Error creating WebGL context.'
And it point out that the GPU used on the server is blacklisted because it's old.

Using wappalyzer and puppeteer in node.js

I am trying to build a scraper to monitor web projects automatically.
So far so good, the script is running, but now I want to add a feature that automatically analyses what libraries I used in the projects. The most powerful script for this job is wappalyser. They have a node package (https://www.npmjs.com/package/wappalyzer) and it's written that you can use it combined with pupperteer.
I managed to run pupperteer and to log the source code of the sites in the console, but I don't get the right way to pass the source code to the wappalyzer analyse function.
Do you guys have a hint for me?
I tryed this code but a am getting a TypeError: url.split is not a function
function getLibarys(url) {
(async () => {
const browser = await puppeteer.launch({ headless: true });
const page = await browser.newPage();
await page.goto(url);
// get source code with puppeteer
const html = await page.content();
const wappalyzer = new Wappalyzer();
(async function () {
try {
await wappalyzer.init()
// Optionally set additional request headers
const headers = {}
const site = await wappalyzer.open(page, headers)
// Optionally capture and output errors
site.on('error', console.error)
const results = await site.analyze()
console.log(JSON.stringify(results, null, 2))
} catch (error) {
console.error(error)
}
await wappalyzer.destroy()
})()
await browser.close()
})()
}

Fixed it by using the sample code from wappalyzer.
function getLibarys(url) {
const Wappalyzer = require('wappalyzer');
const options = {
debug: false,
delay: 500,
headers: {},
maxDepth: 3,
maxUrls: 10,
maxWait: 5000,
recursive: true,
probe: true,
proxy: false,
userAgent: 'Wappalyzer',
htmlMaxCols: 2000,
htmlMaxRows: 2000,
noScripts: false,
noRedirect: false,
};
const wappalyzer = new Wappalyzer(options)
;(async function() {
try {
await wappalyzer.init()
// Optionally set additional request headers
const headers = {}
const site = await wappalyzer.open(url, headers)
// Optionally capture and output errors
site.on('error', console.error)
const results = await site.analyze()
console.log(JSON.stringify(results, null, 2))
} catch (error) {
console.error(error)
}
await wappalyzer.destroy()
})()
}

I do not know if you still need an answer to this. But this is what a wappalyzer collaborator told me:
Normally you'd run Wappalyzer like this:
const Wappalyzer = require('wappalyzer')
const wappalyzer = new Wappalyzer()
await wappalyzer.init() // Launches a Puppeteer instance
const site = await wappalyzer.open(url)
If you want to use your own browser instance, you can skip wappalyzer.init() and assign the instance to wappalyzer.browser:
const Wappalyzer = require('wappalyzer')
const wappalyzer = new Wappalyzer()
wappalyzer.browser = await puppeteer.launch() // Use your own Puppeteer launch logic
const site = await wappalyzer.open(url)
You can find the discussion here.
Hope this helps.

Puppeteer not working as excepted (pause/freeze when browser not in focus)

I'm using Puppeteer.js to crawl some URL. I'm using the default Chromium browser of Puppeteer.All is working well, but the problem is, that when I run the crawling script, and doing other things in the background and the focus is no longer on the Chromium browser of Puppeteer, it's not working: waiting for elements way too long, and abort operations, or in other words: puppeteer is paused (or freeze).
P.S, I'm also using puppeteer-extra and puppeteer-extra-plugin-stealth NPM packages for advance options.
Here is how I create the browser and the page:
async initiateCrawl(isDisableAsserts) {
// Set the browser.
this.isPlannedClose = false;
const browser = await puppeteerExtra.launch({
headless: false,
args: [
'--no-sandbox',
'--disable-setuid-sandbox',
'--disable-dev-shm-usage',
'--start-maximized',
'--disable-background-timer-throttling',
'--disable-backgrounding-occluded-windows',
'--disable-renderer-backgrounding'
]
});
const pid = browser.process().pid;
browser.on('disconnected', () => {
systemUtils.killProcess(pid);
if (!this.isPlannedClose) {
systemUtils.exit(Status.BROWSER_CLOSE, Color.RED, 0);
}
});
process.on('SIGINT', () => {
this.close(browser, true);
});
// Set the page and close the first empty tab.
const page = await browser.newPage();
const pages = await browser.pages();
if (pages.length > 1) {
await pages[0].close();
}
await page.setRequestInterception(true);
await page.setJavaScriptEnabled(false);
await page.setDefaultNavigationTimeout(this.timeout);
page.on('request', (request) => {
if (isDisableAsserts && ['image', 'stylesheet', 'font', 'script'].indexOf(request.resourceType()) !== -1) {
request.abort();
} else {
request.continue();
}
});
return {
browser: browser,
page: page
};
}
I already looked at:
https://github.com/puppeteer/puppeteer/issues/3339
https://github.com/GoogleChrome/chrome-launcher/issues/169
https://www.gitmemory.com/issue/GoogleChrome/puppeteer/3339/530620329
Not working solutions:
const session = await page.target().createCDPSession();
await session.send('Page.enable');
await session.send('Page.setWebLifecycleState', {state: 'active'});
const chromeArgs = [
'--disable-background-timer-throttling',
'--disable-backgrounding-occluded-windows',
'--disable-renderer-backgrounding'
];
var ops = {args:[
'--kiosks',
'--disable-background-timer-throttling',
'--disable-backgrounding-occluded-windows',
'--disable-renderer-backgrounding',
'--disable-canvas-aa',
'--disable-2d-canvas-clip-aa',
'--disable-gl-drawing-for-tests',
'--disable-dev-shm-usage',
'--no-zygote',
'--use-gl=desktop',
'--enable-webgl',
'--hide-scrollbars',
'--mute-audio',
'--start-maximized',
'--no-first-run',
'--disable-infobars',
'--disable-breakpad',
'--user-data-dir='+tempFolder,
'--no-sandbox',
'--disable-setuid-sandbox'
], headless: false, timeout:0 };
puppeteer = require('puppeteer');
browser = await puppeteer.launch(ops);
page = await browser.newPage();
Has anyone faced this issue before and have any idea how to solve this? Thanks.

My issue was solved when I updated to the latest puppeteer version (9.0.0).

How to use Puppeteer with Stripe Elements

Been slamming my head against this for a while now and no idea why this is happening.
I'm using react-stripe-elements and trying to write a test using Puppeteer. I simply cannot get Puppeteer to fill in the Card Elements.
I've tried a few approaches
Approach 1
I try to select the input by its name and then any input on the page by its class
await page.$eval('input[name="cardnumber"]')
await page.$eval(".InputElement");
I'm told that there's
Approach 2
I then tried to access the actual frame, my reasoning being that its technically a page with a different origin. Again, nothing happens. Now, strangely, when I try and print out the contents of the frame, nothing happens again.
const cardExpiryFrameHandle = await page.$('iframe[name="__privateStripeFrame5"]')
const cardExpiryFrame = await cardExpiryFrameHandle.contentFrame()
const test = await cardExpiryFrame.content()
console.log(test);
When I console log out cardExpiryFrame, it exists. This should fit the API defined here https://pptr.dev/#?product=Puppeteer&version=v3.3.0&show=api-class-frame, but it absolutely refuses to.
I also added arguments to disable some security features because I tracked down an issue that said that this might be a problem. I do so like this
module.exports = {
server: {
command: `npm run-script start:test:server`,
port: 3000,
launchTimeout: 100000,
debug: true,
args: ['--disable-web-security', '--disable-features=IsolateOrigins,site-per-process'],
},
launch: {
headless: false,
},
}
Approach 3
I then tried to mimic what a human would do and clicked the div and then tried to type out the test card number.
await page.click(getClass(paymentFlowCardInput))
await page.keyboard.type('4242424242424242', { delay: '50' })
Again no luck. No errors.
Now I'm out of ideas - what do I do?

A good solution for this is using tab to switch to the next input. In my test I have an input for the cardholder name and I then tab to the CardElement component.
describe('Test Input', () => {
test('Filling out card payment form', async () => {
let browser = await puppeteer.launch({
headless: false,
slowMo: 100
});
let page = await browser.newPage();
page.emulate({
viewport: {
width: 1280,
height: 1080
},
userAgent: ''
});
await page.goto('http://localhost:3000/asd/payment-requests/50-eur-by-2021-01-15t1200');
await page.waitForSelector('#input-name');
await page.click('input[name=card_name]')
await page.type('input[name=card_name]', 'Joe Bloggs')
await page.keyboard.press("Tab");
await page.keyboard.type(card.number, { delay: 500 })
await page.keyboard.type(card.month, { delay: 50 })
await page.keyboard.type(card.year, { delay: 50 })
await page.keyboard.type(card.cvc, { delay: 50 })
await page.keyboard.type(card.zip, { delay: 50 })
browser.close();
}, 90000);
});

You're likely running into this issue because your test isn't waiting for the CardElement to mount (and finish its animations) or, the animations are slower than your delay. Here's an example of a puppeteer test which takes those transitions into account for your reference: https://github.com/stripe/react-stripe-elements/issues/137#issuecomment-352092164