We Keep Coding

Puppeteer React/NodeJS form submissions automation with a database

So I'm trying to test a recaptcha implementation by writing my own spambot using React and Puppeteer. I got the script ready to do a single form submission after executing the script, but what I'm actually hoping for is to have my script loop through a database with form submission details, and then reiterate every row of the csv file until it's depleted the database.
So far I have the following script:
const puppeteer = require('puppeteer');
// Server Authentication
const username = "username";
const password = 'password';
(async () => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
// Pass server side authentication needed for website when accessing url
await page.authenticate({ username, password});
// Load page with form and take screenshot to confirm page has been reached
await page.goto('https://pagewiththeformimtryingtosubmit.com');
await page.screenshot({path: 'example2.png'});
// Click away cookie pop-up banner, take screenshot after
page.keyboard.press('Escape');
await page.screenshot({path: 'cookiebotcleared.png', fullPage: true});
// Fill in first form fields, screenshot to see if it works
await page.type('#firstname', 'Somename', {delay:500});
await page.type('#lastname', 'Somelastname', {delay:500});
await page.type('#telephone', '123456789', {delay:500});
await page.type('#email_address', 'somename.lastname#gmail.com', {delay:500});
await page.type('#password', 'Chooseapassword', {delay:500});
await page.type('#password-confirmation', 'Chooseapassword', {delay:500});
await page.click('.consent', {delay:500});
await page.click('.submit', {delay: 500});
// Take screenshot after form submission to see if it has worked
await page.screenshot({path: 'formfillout.png', fullPage: true}, {delay: 500});
await browser.close();
})();
What I'm trying to do, is take a CSV with all the data I've randomized, and then have this script run but take elements from the csv for the various form inputs and loop that over.
I've tried working with CSVToJSON in order to process the csv database into objects that I should then be able to use in my code:
const CSVToJSON = require('csvtojson')
CSVToJSON().fromFile('formbot_database.csv')
.then(users => {
console.log(users);
console.log(users.firstname);
}).catch(err => {
console.log(err);
});
Here's where my first troubles start: I want to take the row headers of my database to map them to variables, so I can process those variables within my script. I first tried users.firstname, but when I console log that, it gives me undefined.
If anyone has any suggestion on how I can work this through, that'd be great. I've tried visiting multiple resources but can't figure it out I'm afraid.
Thanks in advance!

Signing into Office with Puppeteer not working?

Whenever I try to use Puppeteer to sign into an Office account, it always says to "Enter a valid email address" on the logging in site (and I know the email is valid, I have tested it). I looked through some other questions, and one of them mentioned that they might have login preventions for Puppeteer, Selenium, etc, though I am not sure. I am able to login if I do it manually.
Here's the code that handles all of that:
var email = "myemail#outlook.com"
await page.evaluate((em)=>{
document.querySelector(`[type="email"]`).value = em;
},email);
await page.screenshot({
path:"./email_input.png"
})
await page.waitForTimeout(1000);
await page.evaluate((em)=>{
var interval = setInterval(() => {
if(document.querySelector(`[type="email"]`).value === em){
document.querySelector(`[type="submit"]`).dispatchEvent(new Event('click'));
clearInterval(interval);
}
});
},email);
await page.screenshot({
path:"./sign_in_email.png"
})
edit to ManuelMB's comment: It was just a check to make sure that the value was actually the email. It still doesn't work without it.

Puppeteer Open Dropdown Menu and then click on [1st 2nd 3rd...] option

https://www.nhtsa.gov/ratings
I have been trying to make puppeteer select an option from the dropdown menu, so I may scrape information from the website above.
1.I'm having issues declaring a correct selector for puppeteer to understand.
What I mean is after telling puppeteer to click "manufacturer" at the end of the paragraph. I can't seem to click( or maybe select??) an option.
The default option in this dropdown menu is select a manufacturer
2.I also would like to know how I may select the 2nd 3rd and 4th option without hard coding it in.
I haven't even begun scraping information /sad ;(
const puppeteer = require('puppeteer');
async function spider() {
try {
let browser = await puppeteer.launch({ headless: false});
let page = await browser.newPage();
await page.goto('https://www.nhtsa.gov/ratings');
await page.click('a[data-target=".manufacturer-search-modal"]');
await page.click('select');
await page.click('option[value="AUDI"]');
} catch(error) {
console.log(error)
await browser.close();
}
}
export default spider

I think there are a few issues. One, sometimes that page has a popup for a survey, which you may need to close first. Two, you need to wait a bit between clicking the .manufacturer-search-modal link and trying to interact with the select box, because the options in the box aren't populated immediately (maybe it's making a request to the server to get the list of options). Three, I think select boxes are a little special and clicking on them doesn't work, but you can use page.select instead. Putting that all together, plus some ugly code for selecting items by number:
async function spider() {
try {
let browser = await puppeteer.launch({ headless: false});
let page = await browser.newPage();
await page.goto('https://www.nhtsa.gov/ratings');
try {
// Give the 'take our survey' box a chance to pop up, and close it if it does
await page.waitFor('.acsCloseButton', { timeout: 1000 });
await page.click('.acsCloseButton')
} catch {
}
await page.click('a[data-target=".manufacturer-search-modal"]');
// wait for the options to be populated
await page.waitFor('option:nth-child(2)');
// search for AUDI
await page.select('select', 'AUDI');
await page.click('.manufacturer-search-submit');
// select third element in drop-down
// there's probably a better way to do this
const options = await page.$$('option');
const properties = await options[2].getProperties();
const value = await properties.get('_value').jsonValue();
await page.select('select', value);
await page.click('.manufacturer-search-submit');
} catch(error) {
console.log(error)
await browser.close();
}
}

how to Ctrl+A page and Input that into a string

I am making a puppeteer program were it checks Google Docs
I want to be able to select all the text in the Google Docs and be able to make it into a string or a variable
I tried to Copy the text in the Google docs But didn't know where to go from there
await page2.keyboard.down('Control');
await page2.keyboard.press('KeyA');
await page2.keyboard.press('KeyC');
await page2.keyboard.up('Control');
I don't know where to go from here

try
await page.keyboard.down('ControlLeft')
await page.keyboard.press('KeyA')
await page.keyboard.press('KeyC');
await page.keyboard.up('ControlLeft')

Puppeteer: How to submit a form?

Using puppeteer, how could you programmatically submit a form? So far I've been able to do this using page.click('.input[type="submit"]') if the form actually includes a submit input. But for forms that don't include a submit input, focusing on the form text input element and using page.press('Enter') doesn't seem to actually cause the form to submit:
const puppeteer = require('puppeteer');
(async() => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto('https://stackoverflow.com/', {waitUntil: 'load'});
console.log(page.url());
// Type our query into the search bar
await page.focus('.js-search-field');
await page.type('puppeteer');
// Submit form
await page.press('Enter');
// Wait for search results page to load
await page.waitForNavigation({waitUntil: 'load'});
console.log('FOUND!', page.url());
// Extract the results from the page
const links = await page.evaluate(() => {
const anchors = Array.from(document.querySelectorAll('.result-link a'));
return anchors.map(anchor => anchor.textContent);
});
console.log(links.join('\n'));
browser.close();
})();

If you are attempting to fill out and submit a login form, you can use the following:
await page.goto('https://www.example.com/login');
await page.type('#username', 'username');
await page.type('#password', 'password');
await page.click('#submit');
await page.waitForNavigation();
console.log('New Page URL:', page.url());

Try this
const form = await page.$('form-selector');
await form.evaluate(form => form.submit());
For v0.11.0 and laters:
await page.$eval('form-selector', form => form.submit());

I was scraping a SPA, and I had to use waitForNetworkIdle since the form submit was not triggering a page navigation event. Instead it submitted data to the server, and updated the DOM of the page which was already loaded.
const [response] = await Promise.all([
page.waitForNetworkIdle(),
page.click('#form-submit-button'),
]);
When to use waitForNetworkIdle
I suspect that if you open a normal web browser, submit the form, and look to see if the page URL has changed or not. If it has not changed, you should use waitForNetworkIdle.
Also, take this advice with a grain of salt, I've only been using puppeteer for an hour.