Hello,
I´m currently working on an Javascript based on Nodejs where I use Puppeteer as an help to scrape the web. As you can read in the title I´m trying to click on a dropdown menu Item, where the dropdown changes if you type in something diffrent. First here is my code:
// Navigate to the Homepage
await page.goto('https://www.futbin.com/');
await page.click('#player_search');
await page.keyboard.type(playerName);
await page.keyboard.press('ArrowDown');
await page.keyboard.press('ArrowDown');
// Create a screenshot
await page.screenshot({
path: 'screenshot.png'
});
So Basically i do the screenshot just for proving that the headless-browser does the right thing.
The website is futbin, if you want to see how their website works and take a look at the inspect, i think that could help.
But my real problem is that normally when you press enter you directly go to the player page (where i want to get). but after my script there comes always the error "no target". So the keyboard.press('Enter') don´t works. Also other Suggestions from SO didn´t work for me as the dropdown isn´t native and don´t hast counting indexes.
I would really appreciate some Suggestions !
at the end i wanted leave the html code from the first row of the dropdown as well, but i never worked so i would appreciate if you´d take a look at the website please !
You were very close, you dont need to use the arrows as you can find the item to click another way. The secret is to add a waitForSelector as the dropdown makes a call to a api endpoint. Also notice the waitForSelector on the final page to render before we take the screenshot.
Therefore just do this:-
const puppeteer = require('puppeteer');
async function run() {
const browser = await puppeteer.launch( {
headless: false
});
const page = await browser.newPage();
await page.goto('https://www.futbin.com/');
await page.type('#player_search', "Dave");
await page.waitForSelector("ul li a[data-id]");
await page.click("ul li a[data-id]");
await page.waitForSelector('#cal');
await page.screenshot( { path: "./dave.png"});
await browser.close();
};
run();
EDIT ADDTIONAL
To select any index use:-
let index = 3;
let selector = "ul li:nth-child(" + index +") a[data-id]"
await page.click(selector);
This goes to the third item in the dropdown. HTH
Related
I have to use PuppeteerSharp to login to one web site.
The web page uses JavaScript to generate dynamic class, so each time when I tried to login, I can see different class name.
So using Puppeteer Recorder is useless, I can't use its code to login.
But I checked its HTML structure, I found it is rather simple: there is one submit form with 3 input fields, the first one is input field for User_Name,
the second input field for Password, the third is button to submit the form.
So nowadays, I can use the following code to login with a lot of keyboard typings. But my code was NOT a good one, as other keyboard actions will make it a mess.
But I can't find any good PuppeteerSharp code to do something like this:
Find all 3 input fields, then type the user name in the first one, type the password in the second one, and click on the third one.
Please advise!
_page = await _bookie_browser.NewPageAsync();
await _page.SetViewportAsync((new ViewPortOptions { Width = 1920, Height = 3938 }));
await _page.GoToAsync("https://www.webpage.com");
await _page.WaitForTimeoutAsync(2000);
await _page.FocusAsync("input");
await _page.Keyboard.TypeAsync("username");
await _page.WaitForTimeoutAsync(1000);
await _page.Keyboard.PressAsync("Tab");
await _page.WaitForTimeoutAsync(1000);
await _page.Keyboard.TypeAsync("password");
await _page.Keyboard.PressAsync("Tab");
await _page.WaitForTimeoutAsync(1000);
await _page.Keyboard.PressAsync("Enter");
await _page.WaitForTimeoutAsync(1000);
https://www.nhtsa.gov/ratings
I have been trying to make puppeteer select an option from the dropdown menu, so I may scrape information from the website above.
1.I'm having issues declaring a correct selector for puppeteer to understand.
What I mean is after telling puppeteer to click "manufacturer" at the end of the paragraph. I can't seem to click( or maybe select??) an option.
The default option in this dropdown menu is select a manufacturer
2.I also would like to know how I may select the 2nd 3rd and 4th option without hard coding it in.
I haven't even begun scraping information /sad ;(
const puppeteer = require('puppeteer');
async function spider() {
try {
let browser = await puppeteer.launch({ headless: false});
let page = await browser.newPage();
await page.goto('https://www.nhtsa.gov/ratings');
await page.click('a[data-target=".manufacturer-search-modal"]');
await page.click('select');
await page.click('option[value="AUDI"]');
} catch(error) {
console.log(error)
await browser.close();
}
}
export default spider
I think there are a few issues. One, sometimes that page has a popup for a survey, which you may need to close first. Two, you need to wait a bit between clicking the .manufacturer-search-modal link and trying to interact with the select box, because the options in the box aren't populated immediately (maybe it's making a request to the server to get the list of options). Three, I think select boxes are a little special and clicking on them doesn't work, but you can use page.select instead. Putting that all together, plus some ugly code for selecting items by number:
async function spider() {
try {
let browser = await puppeteer.launch({ headless: false});
let page = await browser.newPage();
await page.goto('https://www.nhtsa.gov/ratings');
try {
// Give the 'take our survey' box a chance to pop up, and close it if it does
await page.waitFor('.acsCloseButton', { timeout: 1000 });
await page.click('.acsCloseButton')
} catch {
}
await page.click('a[data-target=".manufacturer-search-modal"]');
// wait for the options to be populated
await page.waitFor('option:nth-child(2)');
// search for AUDI
await page.select('select', 'AUDI');
await page.click('.manufacturer-search-submit');
// select third element in drop-down
// there's probably a better way to do this
const options = await page.$$('option');
const properties = await options[2].getProperties();
const value = await properties.get('_value').jsonValue();
await page.select('select', value);
await page.click('.manufacturer-search-submit');
} catch(error) {
console.log(error)
await browser.close();
}
}
I'm trying to automate the verification code sent to an email in mailinator, when I run the test therror is: "TimeoutError: Wait timed out after 35001ms", I'm thinking that is a problem with the async functions but I'm not secure about that.
const emailRow = element(by.className("tr.even.pointer.ng-scope"));
this.setCode = async function() {
let windows = await browser.getAllWindowHandles();
await browser.switchTo().window(windows[1]);
await browser.wait(ExpectedConditions.visibilityOf(emailRow), 50000);
browser.actions().mouseMove(emailRow).click().perform();
await browser.wait(ExpectedConditions.visibilityOf(emailCode), 35000);
}
I also tried this
this.setCode = async function() {
let windows = await browser.getAllWindowHandles();
await browser.switchTo().window(windows[1]);
await browser.wait(ExpectedConditions.elementToBeClickable(emailRow), 50000);
emailRow.click();
await browser.wait(ExpectedConditions.visibilityOf(emailCode), 35000);
}
But I have the same problem, in the screen I can't see that the test perform the click, I put an sleep after the click in the emailRow but doesn't work, in the image there is the page that i want to perform the click.
I think your best bet is to use their api, instead of going to their website and read the email there. In protractor, this is very easy. depending on whether or not you have a premium account with them you can use a public or a private team inbox. In the case of a public inbox, perform something similar to the following:
const checkMailinatorPublicEmail = async () => {
process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0";
let requestUrl = 'https://mailinator.com/api/v2/domains/public/inboxes/your_inbox_name_here';
let responseBody = await fetch(requestUrl);
let responseJson = await responseBody.json();
return responseJson;
}
Now, you have all the email in the inbox in your response body as a json object. To keep this simple, do not use a static public team inbox, instead use a random inbox name so that each time you will only have one email in the inbox and you can parse that email to your needs.
I believe you should try the second approach. In the first approach, you are waiting for an element to be visible that does not guarantee an element is clickable.
By looking at second approach, the code looks fine. My suggestion is to try changing click method like below
browser.executeScript('arguments[0].click()', emailRow.getWebElement());
Hope this will help
Happy coding!
I was trying to query a website: const url = "https://personal.vanguard.com/us/FixedIncomeHome" with the hope to automate some functionality within puppeteer.
I noticed if i create a screen shot: page.screenshot("preclick.png") it will show the page data with tabs. When i try to follow it up with a query, it seems to not return the second tab (denoted by the following selector: a[container="CD"]
const browser = await puppeteer.launch()
const page = await browser.newPage()
await page.goto(url, {waitUntil: 'networkidle2'})
page.screenshot("start.png")
page.evaluate( () => {
document.querySelectorAll("a[container='CD']")[0].click()
})
///...
and i dont really know why this is the case. Ideally, i am trying to click CD and then click an empty search. I noticed that since session ids are tracked, I wanted to do this as a sort of E2E test in order to get the resulting table data.
I see that the Content of tab etc is dynamically loaded, so somehow there is an issue with the page being able to query.
I was attempting something else to see what would occur, waiting for the tag to appear, BUT it would just timeout after 30 seconds:
await page.waitForSelector("a[container='CD']").then( async resolve => {
page.execute( () => document.querySelector("a[container='CD']").click() );
});
I dont know why the screenshot shows the HTML, but when attempting to query for it from within execute it fails. It doesnt make sense to me why this occurs. Ideally, I want to click CD tab, then i want to click Search, then i want to loop through the 20 results in the table.
EDIT I was noticing that evaluate was not querying the component correctly because of an iframe. If i want to develop e2e testing though, i assumed there was a way to somehow get a reference to the button and click it, or simulate a click.*
You can get the iframe from a selector. As the iframe has the ID TWRIFrame, you can wait for that selector, then get the contentFrame from that element.
Once you have the frame, the frame class has almost the same functions as the page class, e.g. click.
Notice that, as that iframe is from other domain, with the --disable-features=site-per-process flag.
const browser = await puppeteer.launch({headless: false, args: ['--disable-features=site-per-process']});
const page = await browser.newPage();
await page.goto('https://personal.vanguard.com/us/FixedIncomeHome', {waitUntil: 'networkidle2'});
await page.screenshot("start.png");
await page.waitForSelector('#TWRIFrame');
const frameElement = await page.$('#TWRIFrame');
const frame = await frameElement.contentFrame();
await frame.click("a[container='CD']");
I am making a puppeteer program were it checks Google Docs
I want to be able to select all the text in the Google Docs and be able to make it into a string or a variable
I tried to Copy the text in the Google docs But didn't know where to go from there
await page2.keyboard.down('Control');
await page2.keyboard.press('KeyA');
await page2.keyboard.press('KeyC');
await page2.keyboard.up('Control');
I don't know where to go from here
try
await page.keyboard.down('ControlLeft')
await page.keyboard.press('KeyA')
await page.keyboard.press('KeyC');
await page.keyboard.up('ControlLeft')