I have read a few different QAs related to this but none seem to be working.
I am trying to target an element (Angular) called mat-radio-button with a class called mat-radio-checked. And then select the inner text.
In Chrome this is easy:
https://i.stack.imgur.com/Ev0iQ.png
https://i.stack.imgur.com/lVoG3.png
To find the first element that matches in Playwright I can do something like:
let test: any = await page.textContent(
"mat-radio-button.mat-radio-checked"
);
console.log(test);
But if I try this:
let test: any = await page.$$(
"mat-radio-button.mat-radio-checked"
);
console.log(test);
console.log(test[0]);
console.log(test[1]);
});
It does not return an array of elements I can select the inner text of.
I need to be able to find all elements with that class so I can use expect to ensure the returned inner text is correct, eg:
expect(test).toBe("Australian Citizen");
I found out the issue was due to the page generating beforehand and the elements were not available. So I added a waitForSelector:
await page.waitForSelector("mat-radio-button");
const elements = await page.$$("mat-radio-button.mat-radio-checked");
console.log(elements.length);
console.log(await elements[0].innerText());
console.log(await elements[1].innerText());
console.log(await elements[2].innerText());
public async validListOfItems(name) {
await this.page.waitForSelector(name, {timeout: 5000});
const itemlists = await this.page.$$(name);
let allitems: string[]=new Array();
for await (const itemlist of itemlists) {
allitems.push(await itemlist.innerText());
}
return allitems;
}
Make a common method and call by sending parameters.
Related
I'm writing a scraper for Skyscanner just for fun. What I'm trying to do is to iterate through the list of all listings, and for each listing, extract the URL.
What I've done so far is getting the listing $("div[class^='FlightsResults_dayViewItems']") which returns
but I'm not sure how to iterate through the returned object and get the URL (/transport/flight/bos...). The pseudo code that I have is
for(listings in $("div[class^='FlightsResults_dayViewItems']")) {
go to class^='EcoTickerWrapper_itineraryContainer'
go to class^='FlightsTicket_container'
go to class^='FlightsTicket_link' and get the href and save in an array
}
How would I go about doing this?
Side-note, I'm using cheerio and jquery.
Update:
I figured out the CSS selector is
$("div[class^='FlightsResults_dayViewItems'] > div:nth-child(at_index_i) > div[class^='EcoTicketWrapper_itineraryContainer'] > div[class^='FlightsTicket_container'] > a[class^='FlightsTicket_link']").href
Now, I'm trying to figure out how to loop through the listing and apply the selector for each listing in the loop.
Also, it seems like not including the div:nth-child(at_index_i) won't work. Is there a way around this?
$("div[class^='FlightsResults_dayViewItems'] > div:nth-child(3) > div[class^='EcoTicketWrapper_itineraryContainer'] > div[class^='FlightsTicket_container'] > [class^='FlightsTicket_link']").attr("href")
"/transport/flights/bos/cun/210301/210331/config/10081-2103010815--32733-0-10803-2103011250|10803-2103311225--31722-1-10081-2103312125?adults=1&adultsv2=1&cabinclass=economy&children=0&childrenv2=&destinationentityid=27540602&inboundaltsenabled=false&infants=0&originentityid=27539525&outboundaltsenabled=false&preferdirects=false&preferflexible=false&ref=home&rtn=1"
$("div[class^='FlightsResults_dayViewItems'] > div[class^='EcoTicketWrapper_itineraryContainer'] > div[class^='FlightsTicket_container'] > [class^='FlightsTicket_link']").attr("href")
undefined
Here's the function to iterate the listings and grab the URLs for each listing.
async function scrapeListingUrl(listingURL) {
try {
const page = await browser.newPage();
await page.goto(listingURL, { waitUntil: "networkidle2" });
// await page.waitForNavigation({ waitUntil: "networkidle2" }); // Wait until page is finished loading before navigating
console.log("Finished loading page.");
const html = await page.evaluate(() => document.body.innerHTML);
fs.writeFileSync("./listing.html", html);
const $ = await cheerio.load(html); // Inject jQuery to easily get content of site more easily compared to using raw js
// Iterate through flight listings
// Note: Using regex to match class containing "FlightsResults_dayViewItems" to get listing since actual class name contains nonsense string appended to end.
const bookingURLs = $('a[class*="FlightsTicket_link"]')
.map((i, elem) => console.log(elem.href))
.get();
console.log(bookingURLs);
return bookingURLs;
} catch (error) {
console.log("Scrape flight url failed.");
console.log(error);
}
}
Using map()
const hrefs = $(selector).map((i, elem) => elem.href).get()
Looking at the code you are not using jQuery so above does not work. So you just need to use a basic selector that matches part of the class with querySelectorAll. And map is used to grab the hrefs.
const links = [...document.querySelectorAll('a[class*="FlightsTicket_link"]')]
.map(l=>l.href)
I have a HTML page that has many div elements, each one with the following structure (the input id and name changes):
<div class="item">
<div class="box">
<div class="img-block">
<label for="check-11">
<input id="check-11" name="result11" type="checkbox">
<span class="fake-input"></span>
</label>
</div>
</div>
</div>
I want to use puppeteer to get all the span with the 'fake-input' class and click on them.
The problem is that it never works, no matter what I try.
In every attempt the start is the same:
(async () => {
const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();
await page.goto(baseUrl, { waitUntil: 'networkidle2' });
// FETCHING AND CLICKING
}();
I tried many things:
1:
await page.waitForSelector('span.fake-input');
await page.click('span.fake-input');
2:
await page.waitForSelector('span.fake-input')
.then(()=>{
console.log(`clicked!`);
page.click('span.fake-input')
3:
const spans = await page.evaluate(() => {
return Array.from(document.querySelectorAll('span'), el => el.textContent)
})
console.log('spans', spans)
for (let index = 0; index < 7; index++) {
const element = spans[index];
await page.click('span')
}'=
4:
await page.evaluate(selector=>{
return document.querySelector(selector).click();
},'span.fake-input)
console.log('clicked');
In every solution the page fails to get anything at all (either return null or undefined, so the error is "click" is not a funciton in null) or it fails with the error "Node is either not visible or not an HTMLElement".
No matter the error, in any case I fail to fetch all the spans, and click on them.
Can anyone tell me what I'm doing wrong?
Use page.$$ to return multiple elements (equivalent of document.querySelectorAll). Use page.$ to return a single element (equivalent of document.querySelector).
If you want to extract a certain value from a group of elements, use page.$$eval and page.$eval for a single element.
e.g. return elementHandle to script
const spans = await page.$$('div#item label .fake-input')
spans.forEach(span=>span.click())
If you extracting a value from an element, pass a callback to it that returns what you need to extract
e.g.
const spanTexts = page.$$eval('div#item label .fake-input', spans => {
spans.map(span=>span.innerText)
})
console.log(spanTexts)
I should add that page.$$eval and page.$eval executes your callback in the browser context.
var obj = document.querySelectorAll("span.fake-input");
for(var i=0;i<obj.length;i++){
obj[i].click();
}
Vanilla JavaScript would work much easier
I'm getting a list of 30 items from the code:
const boxes = await page.evaluate(() => {
return document.querySelectorAll("DIV.a-row.dealContainer.dealTile")
})
console.log(boxes);
The result
{ '0': {},
'1': {},
'2': {},
....
'28': {},
'29': {} }
I have the need to see the html of the elements.
But every property I tried of boxes is simply undefined. I tried length, innerHTML, 'innerText` and some other.
I am sure of box really containing something because puppeteer's screenshot shows the content before I start to 'browse' the content of the page
What am I doing wrong?
There are multiple ways to do this:
Use page.$$eval to execute the selector and return the result in one step.
Use page.evaluate to get the attributes after querying the elements.
Code sample for page.$$eval
const htmls = await page.$$eval('selector', el => el.innerHTML);
Code sample for page.evaluate
const singleBox = boxes[0];
const html = await page.evaluate(el => el.innerHTML, singleBox);
I just started using testcafe, so far I found the documentation helpful to do my test and automate my E2E.
I want to assert if a value exists in a td like this:
async checkBankAccount(accountNumber, currencyCode){
const formatedAccount = formatBankAccount(accountNumber, currencyCode);
console.log(formatedAccount);
await t
.expect(Selector('td').withText(formatedAccount).innerText).eql(formatBankAccount);
}
I am having the following error:
An assertion method is not specified.
I want to assert if it exists a td in my HTML that contains the text from formatedAccount.
Thanks
Use the exists property to check if an element is available.
async checkBankAccount(accountNumber, currencyCode){
const formatedAccount = formatBankAccount(accountNumber, currencyCode);
console.log(formatedAccount);
await t
.expect(Selector('td').withText(formatedAccount).exists).ok();
}
You may use this to assert if your td contains the text coming from formatedAccount as follows:
test ('test..',
async t => {
const field = Selector('td');
const text = field.textContent;
await t
.expect(text).contains(formatedAccount)
});
How can I get a list of elements with a certain class name in javascript with selenium?
I am searching for any elements with the class message_body. I want a array containing all of the elements that have that class.
driver.findElements(By.className("message_body")) doesn't work, it seems to return something else.
How can I get this list?
Here is an example to get the text from a list of elements:
driver.findElements(By.className("message_body")).then(function(elements){
elements.forEach(function (element) {
element.getText().then(function(text){
console.log(text);
});
});
});
So, I'm using an older version of Selenium, v2.47.1, but something I used when driver.findElements(By.className("someClass")) wasn't sufficient was driver.findElements(By.xpath("/path/to/[#class='someClass']")) . This will return a List<WebElement>. If I remember correctly, By.xpath is a little slower than some of the other options on some browsers, but not by a whole lot....
First you need to get the elements collection:
public async getInstalledApps(): Promise<any[]> {
const appsList: WebComponent = this.browser.find(By.css(`div .icons__title`));
return await appsList.getElements();
}
Then, using the function above you can do anything, for example get the text property and save them. For example if it's a group of Apps button and you want to get a names array of them:
public async getInstalledAppsList(): Promise<string[]> {
const appsList: string[] = [];
let app: string = '';
(await this.getInstalledApps()).forEach(async element => {
await Promise.resolve(element).then(async (text: any) => {
app = await (await text.getText());
appsList.push(app);
});
});
return appsList;
}