I'm testing Apify web scraper and i am able to see the results but it's displayed in one row. I'm iterating contents to grab titles from this page - ... and pushing it to array and finally returning array. I would like to see the titles returned in separate rows. Would really appreciate if someone could point me to right direction.
Here is my page function:
async function pageFunction(context) {
// jQuery is handy for finding DOM elements and extracting data from them.
// To use it, make sure to enable the "Inject jQuery" option.
const $ = context.jQuery;
var results = [];
$('.ms-srch-group-content').each(function(){
results.push({
title: $(this).find('.ms-srch-item-link').text().trim(),
date: $(this).find('.soi-news-attributes').text().trim(),
});
});
return results;
}
Here is the result:
...
As you can see in the result screenshot, all the titles are displayed in one row.
The problem here is that you're trying to iterate over the element itself, while what you're trying to achieve would require to move down by one level.
That said - you would only need to change .ms-srch-group-content to .ms-srch-group-content > div[name=Item] and it would work as expected.
Looks like .ms-srch-group-content is selector for group content, so you receive one record not an array.
I guess you need to move down to the next div - .ms-srch-item
$('.ms-srch-item').each(function(){
Related
I have a rough idea on how to do this, but it doesn't seem to be working too well.
What I have already achieved is pulling all of the data necessary that needs ordered. What I need is a way to take all of that information and order it from the highest number to the lowest number, and then display that in a single embed - without the use of adding more fields. Ideally it should look something like the image included, except inside an embed. I want to be able to loop this so that it automatically updates the message every X amount of seconds with a message edit.
Each row is ordered from #1 to #20 with #1 having the most Points. This is in the [0123] bit.
The code used to select data from the table is:
const [team, teamd, teame] = await pool.query("SELECT * FROM `performancetracker`.`leaderboard`");
The columns I have use for are TeamName and Points.
I've done something similar with the following code:
Object.keys(check).forEach(function(key) {
var row = check[key];
let name = row.TeamName
embed1.addField(`Team:`, `${name}`, true)
})
However, this adds fields to the embed, which I don't want. I'm not too sure how to go about creating an array or object that I can add to and edit later in the code while maintaining the ability to add it as a field in an embed. I'm not fluent with JavaScript, I'm still learning new things and finding new challenges.
I'm not sure I can help you with the updating part because I don't fully understand the question, but you can do this for displaying the embed how you want:
let data;
Object.keys(check).forEach(() => {
var row = check[key];
let name = row.TeamName
data += `Team: ${name}\n`
})
embed1.addDescription(data)
I haven't tested this code, but it should work.
I have a Discord bot that scrapes web data from inside an unordered list, the amount of list items in the ul can vary week by week from a minimum of 5 to a peak of 20.
I use this code to scrape the data (node.js and Cheerio):
$('li').eq(65).siblings().each(function(i, elem) {
details[i] = $(this).text().replace(/\*/g, ' ').split('/').join('or').trim();
});
My problem is because the number of list item elements can change on a weekly basis I never know how many addField objects to use when rendering the embed to Discord screen, so I can't add them by array index position as I would either have null values or not enough fields, I need to be able to generate addField objects so that each of the array index's returned is inside its own addField.
The embed is rendered with this code:
const embed = new Discord.RichEmbed()
.setColor(0xf1c40f)
.setTitle('Listing For The Week:')
.addField('Items', `${details}`)
.setTimestamp();
client.channels.get('546033223172620288').send({ embed });
This just displays the entire scraped data inside one field and it looks horrible as you can imagine, which is why I want to iterate the data into seperate .addFields.
I'm learning js so what do I have to do to get my desired outcome? I'm looking at doing something like this:
for (let i = 0; i < details.length; i++) {
.addField('ITEM:', details);
}
but I don't have the knowledge how to work with the .addField object and keep getting errors and finding that google isn't helping me find out how to approach this with the dot notation on addField inside a for loop.
You're one the right track I suppose. You can't simply have a "random" .addField() out there like that though. Simply utilize your existing embed variable within the loop - embed.addField('ITEM', details)
So, I really love this example from Jake Zieve shown here: https://bl.ocks.org/jjzieve/a743242f46321491a950
Basically, on search for a term, the path to that node is highlighted. I would like to accomplish something similar but with the following caveats:
I would like to stay in D3 v4.
I'm concerned about cases where the path doesn't clear out on next node pick OR what happens when there are two nodes of the same
name (I would ideally like to highlight all paths)
I would like to AVOID using JQuery
Given a set search term (assume you're already getting the string from somewhere) I know I need to make use of the following lines specifically (you can see my stream of consciousness in the comments) but I'm just not quite sure where to start.
// Returns array of link objects between nodes.
var links1 = root.descendants().slice(1); //slice to get rid of company.
console.log(links1); //okay, this one is nice because it gives a depth number, this describes the actual link info, including the value, which I am setting link width on.
var links2 = root.links(); // to get objects with source and target properties. From here, I can pull in the parent name from a selected target, then iterate again back up until I get to source. Problem: what if I have TWO of the same named nodes???
console.log(links2);
Thoughts on this? I'll keep trying on my own, but I keep hitting roadblocks. My code can be found here: https://jsfiddle.net/KateJean/7o3suadx/
[UPDATE]
I was able to add a filter to the links2 to call back a specific entry. See
For example:
var searchTerm = "UX Designer"
var match = links2.filter(el => el.target.data.name === searchTerm); //full entry
console.log(match);
This single entry gives me all associated info, including the full list of all points back up to "COMPANY"
So, I can GET the data. I think the best way to accomplish what I want is to somehow add a class to each of these elements and then style on that "active" class.
Thank you!
So using a list of ids, I'm trying to get data for each element in the database and add that element to a list. At the end, I'm displaying it on an HTML page. How would I do that? Right now, since it executes the queries async, the list will be empty after running the code to execute the query and add the result to a list.
Lets assume you have a table called teams that contains a row for each team, with columns like id, games, total_points, wins, and losses.
Lets assume you have an array of team id. You want to query the database for those records, then do some things with them before returning them to the front end as json.
This is how i understand your problem. If this isn't the case, please clarify in a comment on this answer.
Possible solution:
Do it all in one query. I'm going to use my favorite mysql library, knexjs.
var columns = ['id', 'games', 'total_points', 'wins', 'loses'];
function getTeams(ids){
return knex.select(columns).where('id', 'in', ids)
.then((teams) => {
//do whatever with array of teams
return teams;
}
}
It's pretty much as simple as that.
I don't think this is strictly infinite scrolling but it was the best i could think of that compares to what i am seeing.
Anyway, we are using ng-grid to show data in a table. We have roughly 170 items (rows) to display. when we use ng-grid it creates a repeater. When i inspect this repeater from the browser its limited to 35 items and, as you scroll down the list, you start to lose the top rows from the dom and new rows are added at the bottom etc (hence why i don't think its strictly infinite scrolling as that usually just adds more rows)
Just so I'm clear there is always 35 'ng-repeat=row in rendered rows' elements in the dom no matter how far you have scrolled down.
This is great until it comes to testing. I need to get the text for every item in the list, but using element.all(by.binding('item.name')) or by.repeater or by.css doesn't help as there is only ever 35 items present on the page.
Now to my question, how can i make it so that i can grab all 170 items as an object that i can then iterate through to grab the text of and store as an array?
on other pages where we have less than 35 items iv just used the binding to create an object, then using async.js to go over each row and get text (see below for an example, it is modified extract i know it probably wouldn't work as it is, its just to give you reference)
//column data contains only 35 rows, i need all 170.
var columnData = element.all(by.binding('row.entity.name'))
, colDataTextArr = []
//prevOrderArray gets created elsewhere
, prevOrderArray = ['item1', 'item2'... 'item 169', 'item 170'];
function(columnData, colDataTextArr, prevOrderArray){
columnData.then(function(colData){
//using async to go over each row
async.eachSeries(colData, function(colDataRow, nRow){
//get the text for the current async row
colDataRow.getText().then(function(colDataText){
//add each rows text to an array
colDataTextArr.push(colDataText);
nRow()
});
}, function(err){
if(err){
console.log('Failed to process a row')
}else{
//perform the expect
return expect(colDataTextArr).toEqual(prevOrderArray);
}
});
});
}
As an aside, I am aware that iterating through 170 rows and storing the text in an array isn't very efficient so if there is a better way of doing that as well I'm open to suggestions.
I am fairly new to JavaScript and web testing so if im not making sense because I'm using wrong terminology or whatever let me know and i'll try and explain more clearly.
I think it is an overkill to test all the rows in the grid. I guess it would be sufficient to test that you get values for the first few rows and then, if you absolutely need to test all the elements, use an evaluate().
http://angular.github.io/protractor/#/api?view=ElementFinder.prototype.evaluate
Unfortunately there is no code snippet in the api page, but it would look something like this:
// Grab an element where you want to evaluate an angular expression
element(by.css('grid-selector')).evaluate('rows').then(function(rows){
// This will give you the value of the rows scope variable bound to the element.
expect(rows[169].name).toEqual('some name');
});
Let me know if it works.