So I have been a mess for days. I am scraping a website for particular information. The problem is that the website has two css classes but with an identical name. I want to use the link and text from the first css class. Attached is the image of what I have. I want to only use the href values from 1 and not the ones from the two "regions".
const cheerio = require('cheerio');
const axios = require("axios");
const siteUrl = "https://worldpostalcode.com/nigeria/abia/";
const fetchData = async () => {
const result = await axios.get(siteUrl);
return cheerio.load(result.data);
};
const getData = async (html) => {
const stateList = []
const $ = await fetchData();
const stateUrl = $('.regions',html);
//console.log(stateUrl.length)
console.log(stateUrl.length)
for (let index = 0; index < 1; index++) {
let firstRegion = $(stateUrl[index],'a')
stateList.push(firstRegion)
}
console.log(stateList)
}
getData()
Help please
I would use the previous h2 text:
$('h2:contains(Regions) + div.regions a')
Related
I've read multiple posts on here about reading in table data, however, every post is how to read and use that data within a single test function. How do I read a table in, and use it in any test function within a test spec?
If I write this into a test function it works and gives back all the info correctly:
import { sAdmin } from "..authentication";
import TablePage from "../TablePage";
const tablePage = new TablePage();
fixture `A fixture`
.page`https://subdomain.example.com/#/table`
.beforeEach( async t => {
await t
.resizeWindow(1284, 722)
.useRole(sAdmin);
});
// this will work, but this won't share context with any other test function in the spec
// this should be initialized at the start to be able to verify search clears
test(`A test`, async t => {
const tableColumn = await tablePage.woNumber;
const tableCount = await tablePage.woNumber.count;
const cellData = [];
for (let i = 0; i < tableCount; i++) {
const text = await tableColumn.nth(i).innerText;
cellData.push(text);
}
console.log('in spec - tableCount', tableCount);
console.log('in spec - cellData', cellData);
});
The output of both console logs is correct:
in spec - tableCount 4
in spec - cellData [ '0007', '0003', '0005', '0006' ]
I've tried an async function in my test spec, and in my page object model (POM). Async function won't work in my spec unless it's within a test function. The one in the POM works, it'll get called, however i can't do const tableCount = await tablePage.woNumber.count; it will yell at me that I can't use a selector like that. It's because of the modifier .count. I adjusted the .count to be within the for loop but that just returned undefined or other data that didn't help.
Example of the async function in my page object model (TablePage)
async rowCount() {
const tableColumn = await tablePage.fooSelector;
const tableCount = await tablePage.fooSelector;
const cellData = [];
for (let i = 0; i < tableCount.count; i++) {
const text = await tableColumn.nth(i).innerText;
cellData.push(text);
}
console.log('in page - tableColumn', tableColumn);
console.log('in page - tableCount', tableCount);
console.log('in page - cellData', cellData);
return tableCount;
};
It's called with this in my spec file, not sure where to call it though:
const count = tablePage.rowCount();
I need this to run after the page has loaded to grab the cell data, and allow me to share context across all tests within this spec file at the very minimum. I'd prefer to put it in my POM, so it can be used elsewhere in other tests. But I'd settle for it working in my test spec without it being in a test function so it can be shared across all tests in the spec file.
I've tried to do a fixture context, but that also had issues and returned undefined. Here is a before with context that I tried, that didn't work.
.before( async ctx => {
const tableColumn = await tablePage.fooSelector;
const tableCount = await tablePage.fooSelector;
for (let i = 0; i < tableCount.count; i++) {
const text = await tableColumn.nth(i).innerText;
ctx.cellData.push(text);
}
// console.log('in spec - tableCount', tableCount);
// console.log('in in spec - cellData', cellData);
})
These console logs return undefined, or objects instead of the text.
Any help would be greatly appreciated. Here are resources I referenced already:
TestCafe - Storing results of Selector in variable
How do I can get a text of all the cells of the table using testcafe
Testcafe get text from element
https://testcafe.io/documentation/402670/reference/test-api/domnodestate
EDIT: I'm still looking for a way to share context of the data I get, I wasn't able to return the data back to the test spec. Maybe if I do more tinkering I can share the values I've obtained.
Here is my solution that doesn't share context, but it does let me prevent code reuse in every test function.
Page Object Model
import { Selector, t } from "testcafe";
class TablePage{
constructor() {
this.searchInput = Selector('#searchInput');
this.tableCount = Selector('.class-selector');
};
async validateSearchResults(selector, searchText) {
await t
.typeText(this.searchInput, searchText)
.pressKey('enter');
const rowCount = await this.tableCount.count;
let searchResults = []
for (let i = 0; i < rowCount; i++) {
let text = await selector.nth(i).innerText;
searchResults.push(text);
await t.expect(searchResults[i]).contains(searchText);
}
};
}
export default TablePage;
Spec File
import { sAdmin } from "..authentication";
import TablePage from "../TablePage";
const tablePage = new TablePage();
fixture `Test search functionality`
.page`https://examplepage.com`
.beforeEach( async t => {
await t
.useRole(sAdmin)
});
test(`User can search via order number`, async t => {
await tablePage.validateSearchResults(tablePage.tableCount, 'foo');
});
The const tableCount = await tablePage.selector.count; looks correct and should work. Also, it's necessary to call the async method with await keyword:
const count = await tablePage.rowCount();
Here is an example of a similar approach:
table-page.js:
import { Selector } from 'testcafe';
export class TablePage {
constructor () {
this.tableCells = Selector('#ContentHolder_grid_DXDataRow0 td');
this.cellData = [];
this.cellCount = 0;
}
async initCellData () {
this.cellCount = await this.tableCells.count;
for (let i = 0; i < this.cellCount; i++) {
const text = await this.tableCells.nth(i).innerText;
this.cellData.push(text);
}
}
}
test.js:
import { TablePage } from './table-page';
let page = null;
fixture `New Fixture`
.page `https://demos.devexpress.com/ASPxGridViewDemos/DataBinding/QueryBuilderControl.aspx`
.beforeEach(async () => {
page = new TablePage();
await page.initCellData();
});
test(`New Test`, async t => {
console.log('cells count: ', page.cellCount);
console.log('cells data: ', page.cellData);
});
I am working on a simple restaurant web app which uses a mongo-db database to store the menu items. My issue is that I have a client js file that will use a routing function that then accesses the database to return all the menu items of a certain restaurant. My issue is that my endpoint for the url isn't being recognized:
Client.js
function readMenu(rest){
(async () => {
// const newURL = url + "/menus/"+rest
const resp = await fetch(url+"/menus/"+rest)
const j = await resp.json();
itemlist = j["items"]
var element = document.getElementById("menu")
var i;
for (i = 0; i < itemlist.length; i++) {
var para = document.createElement("p")
item = itemList[i]
text = item["name"]+" | "+item["cost"]+" | "+item["descr"] +"<br>";
var node = document.createTextNode(text)
para.appendChild(node)
element.appendChild(para)
}
})
}
Server-routing.ts (Routings):
this.router.get("/menus", this.getResturants.bind(this))
this.router.post("/menus", this.addResturaunt.bind(this))
this.router.get("/menus/:rest", this.getResturauntItems.bind(this))
this.router.delete("/menus/:rest",this.deleteResturaunt.bind(this))
this.router.get("/menus/:rest/:item",[this.errorHandler.bind(this),this.getItem.bind(this)])
this.router.post("/menus/:rest",this.addItem.bind(this))
this.router.delete("/menus/:rest/:item",this.deleteItem.bind(this))
Server-routing.ts (function):
public async getResturauntItems(request, response) : Promise<void> {
console.log("Getting Restaurant Items")
let rest = request.params.rest
let obj = await this.theDatabase.getResturauntItems(rest)
console.log(obj)
response.status(201).send(JSON.stringify(obj))
response.end()
}
So, what should happen is a button calls readMenu(), it then makes a GET fetch request to localhost:8080/api/menus/ and then the menu items from the collection should be returned. The issue is that when I click the button, nothing happens. I know it is not being redirected to some other function as they all have "console.log()" to keep track of them and none of them where called. I used the "inspect" tool to see if the request was being sent or received anywhere and nothing. I am unsure of what the issue happens to be. If anyone can help, it would be really appreciated.
you just never called your function, you declared the async function inside your function but never called it.
function readMenu(rest){
(async () => {
// const newURL = url + "/menus/"+rest
const resp = await fetch(url+"/menus/"+rest)
const j = await resp.json();
itemlist = j["items"]
var element = document.getElementById("menu")
var i;
for (i = 0; i < itemlist.length; i++) {
var para = document.createElement("p")
item = itemList[i]
text = item["name"]+" | "+item["cost"]+" | "+item["descr"] +"<br>";
var node = document.createTextNode(text)
para.appendChild(node)
element.appendChild(para)
}
})();
}
you need to add () after creating the functions to call it.
When taking screenshots using puppeteer, dynamic elements with the .menu__link class are required to change innerHTML to a stub.
I use BackstopJs puppet/onReady.js
When I try this, only the first element on the page is replaced:
module.exports = async (page) => {
const myLocalValue = "Test";
const tweets = await page.$$('.menu__link');
for (const tweet of tweets) {
await page.$eval('.menu__link', (el, value) => el.innerHTML = value, myLocalValue)
}
};
And this code does not work at all:
module.exports = async (page) => {
const myLocalValue = "Test";
const tweets = await page.$$('.menu__link');
for (const tweet of tweets) {
await page.$eval(tweet, (el, value) => el.innerHTML = value, myLocalValue)
}
};
Please tell me how to replace innerHTML on the entire page for all .menu__link using puppeteer?
You can use $$eval
await page.$$eval('. menu__link', (links, value) => links.forEach(el => el.innerHTML = value), 'myLocalValue');
this question is probably really easy but I kind of got stuck with Firestore. It's my first project with firestore + js and I'm trying to receive and display data from my database. I have a feeling that I am doing something really stupid because it seems like I am looping through my data and overriding it that's why I can see only one article element even though there should be 2 fetched from the database.
document.addEventListener('DOMContentLoaded', event => {
const app = firebase.app();
const db = firebase.firestore();
const myProducts = db.collection('products');
myProducts.onSnapshot(products => {
const productsContainer = document.querySelector('#products__container');
products.forEach(doc => {
data = doc.data();
console.log(data);
let productMarkup = `<article id="${doc.id}">
<h4>${data.name}</h4>
<p>${data.price}</p>
</article>`;
productsContainer.innerHTML = productMarkup;
console.log(productMarkup);
});
});
});
After Alex suggestion, I decided to take the different approach to create DOM elements
const db = firebase.firestore();
const myProducts = db.collection('products');
const productsContainer = document.querySelector('#products__container');
function renderProduct(doc) {
const docFrag = document.createDocumentFragment();
let article = document.createElement('article');
let productName = document.createElement('h4');
let productPrice = document.createElement('p');
article.setAttribute('id', doc.id);
productName.textContent = doc.data().name;
productPrice.textContent = doc.data().price;
docFrag.appendChild(productName);
docFrag.appendChild(productPrice);
article.appendChild(docFrag);
productsContainer.appendChild(article);
}
myProducts.onSnapshot(products => {
products.forEach(doc => {
data = doc.data();
console.log(data);
renderProduct(doc);
});
});
});```
So I'm trying to crawl a site using Puppeteer. All the data I'm looking to grab is in multiple tables. Specifically, I'm trying to grab the data from a single table. I was able to grab the specific table using a very verbose .querySelector(table.myclass ~ table.myclass), so now my issue is, my code is grabbing the first item of each table (starting from the correct table, which is the 2nd table), but I can't find a way to get it to just grab all the data in only the 2nd table.
const puppeteer = require('puppeteer');
const myUrl = "https://coolurl.com";
(async () => {
const browser = await puppeteer.launch({
headless: true
});
const page = (await browser.pages())[0];
await page.setViewport({
width: 1920,
height: 926
});
await page.goto(myUrl);
let gameData = await page.evaluate(() => {
let games = [];
let gamesElms = document.querySelectorAll('table.myclass ~ table.myclass');
gamesElms.forEach((gameelement) => {
let gameJson = {};
try {
gameJson.name = gameelement.querySelector('.myclass2').textContent;
} catch (exception) {
console.warn(exception);
}
games.push(gameJson);
});
return games;
})
console.log(gameData);
browser.close();
})();
You can use either of the following methods to select the second table:
let gamesElms = document.querySelectorAll('table.myclass')[1];
let gamesElms = document.querySelector('table.myclass:nth-child(2)');
Additionally, you can use the example below to push all of the data from the table to an array:
let games = Array.from(document.querySelectorAll('table.myclass:nth-child(2) tr'), e => {
return Array.from(e.querySelectorAll('th, td'), e => e.textContent);
});
// console.log(games[rowNum][cellNum]); <-- textContent