We Keep Coding

How to use JavaScript to get all element from a dynamic scroll list?

Like the title said, how do I get all elements from a scroll div? The elements in the scroll list are loaded and destroyed dynamically.
I tried to crawl all course names from this website:
https://public.enroll.wisc.edu/search?term=1204
The code below only works for one time:
let list = document.getElementsByClassName('md-virtual-repeat-scroller')[0]
let childs = document.getElementsByClassName("result__name")
console.log(childs[0].innerText)
However, if I do this, I will get the same result for 10 times:
let list = document.getElementsByClassName('md-virtual-repeat-scroller')[0]
for(let i = 0; i < 10; i++) {
let childs = document.getElementsByClassName("result__name")
for(let j = 0; j < childs.length; j++) {
console.log(childs[j].innerText)
}
// scroll by 1000px every time
list.scrollBy(0, 1000)
}
I don't know what's the problem. Is it because that scrollBy() works asynchronously? But I tried to use async and await. It still doesn't work.

Give more information in less words as a possible. Many problems could be related to browser and its version, for example. How is this script called? Are you giving commands via browser console? Have you done a copy of the site and performed some modification on it? It's hard to understand the problem in a realistic level.
Tip: Avoiding use innerText. It's slower and is supported in many browsers only for compability to scripts written to old versions of IE. (I don't know why so many examples in internet use it as first option). User textContent instead.
It's always good to test the returned value of a function/methods - specially during the development of the program.
Never ask to the StackOverFlow community (and to any other) to write progams for you!
You question "how do I get all elements from a scroll div?" is so "loose". scroll div? The answer to this, independently to the "type of div" (and tag!) would be found below.
Your code seems to be no sense in order to do what you want. Why iterate from 0 to 10?
Look at this snipet. I think it will help you
const list = document.getElementsByClassName('md-virtual-repeat-scroller')[0];// if there is no intention to reassign it. Use [0] if you are sure it's the first element of this collection
let childs = list.getElementsByClassName("result__name"); // get only elements inside the first variable!
Use the iterator of the variable.
for(item of childs)
{
/*code*/
}
I am sure you will achieve your goals!
And never suggest us (Community) to code for you or even to resolve your problem. This sound very agressive! To you too! I'm sure.

I solved my problem by reading this article:https://intoli.com/blog/scrape-infinite-scroll/
The reason why I kept getting the same elements is that scrollBy() works asynchronously, so I have to wait then evaluate the page again. I am using puppeteer by the way.
please read the article, super helpful.

Optimizing jQuery selector / addBack() when dealing with a large collection

I use jQuery to intentionally remove css classes from elements in a potentially large html table. See below for an explanation why I am doing that.
Currently I am doing it like this:
var tableElements = $("#TreeListElemente").find("*").addBack();
tableElements.removeClass("dxtl dxtl__B2 dxtl__B0 dxtlSelectionCell dxtlHeader dxtl__B3 dxtlControl dx-wrap dxtl__IM dxeHyperlink");
The table sometimes is large and has many elements. I would like to speed up the page load / DOM manipulation.
The IE's built-in Javascript profiler tells me that especially the .addBack() is slow. It seems to do some kind of sorting, which is totally unnecessary to my use case. Could I get rid of that? Is there another way to include the selected element itself, besides addBack()?
IE javascript profiler: Execution time for a collection of about 60000 elements. The inclusive times are in the third column
Or is there another, more efficient way to remove classes from a large set of elements, with selecting an element, itself, and all children?
Note: Why am I doing this: I am using the DevXpress TreeList Component which comes with it's own styling. There is no easy way to "unstyle it" on the server side, thus I chose to do that client-side, the way demonstrated above. In the end, I am selecting the TreeList, all child elements, and remove the relevant css classes from them.
Update/Solution 1
I have successfully implemented the solution proposed by Frédéric Hamidi an got quite an improvement:
IE javascript profiler: Execution time for a collection of about 60000 elements, using the proposal by Frederic. The inclusive times are in the third column
The time needed for the addBack() operation are just gone, remaining just the other stuff. This means an improvement by more than factor 4 overall. Yay!
Update/Solution 2
I have also implemented the solution proposed by A. Wolff and got a slight additional improvement:
IE javascript profiler: Execution time for a collection of about 60000 elements, using the proposal by A. Wolff. The inclusive times are in the third column
The time needed for the find() operation is gone, remaining just the other stuff again. This means an slight improvement of some 10s of milliseconds on my machine. Cool!
This is the solution I am using now:
$("#TreeListElemente, #TreeListElemente [class]").removeClass("dxtl dxtl__B2 dxtl__B0 dxtlSelectionCell dxtlHeader dxtl__B3 dxtlControl dx-wrap dxtl__IM dxeHyperlink");

addBack() does perform a sort to put the matched elements in document order. The easy alternative, add(), does the exact same thing, so it won't solve your problem.
However, the documentation is helpful enough to provide a solution:
To create a jQuery object with elements in a well-defined order and
without sorting overhead, use the $(array_of_DOM_elements) signature.
Therefore, to avoid that overhead, you can write:
var ancestor = $("#TreeListElemente"),
tableElements = $(ancestor.find("*").get().concat(ancestor[0]));
get() and concat() end up building two arrays under the hood, though, so it will affect performance. The end result may be faster than your current approach, depending on the number of elements you match.

The relevant selector to select element with ID TreeListElemente and all its descendants would be:
"#TreeListElemente, #TreeListElemente *"
Now you could filter out descendants having class:
"#TreeListElemente, #TreeListElemente [class]"
So it would give:
$("#TreeListElemente, #TreeListElemente [class]").removeClass("dxtl dxtl__B2 dxtl__B0 dxtlSelectionCell dxtlHeader dxtl__B3 dxtlControl dx-wrap dxtl__IM dxeHyperlink");

Here's a thought:
function deClassify(jq, classes) {
var remove = classes.join(' ');
jq.find('.' + classes.join(',.')).removeClass(remove);
jq.removeClass(remove);
}
deClassify($('.keepme'), ['remove', 'remove2', 'remove3']);
.remove, .remove2, .remove3 {
color: red;
}
.keepme, .keepme2 {
font-weight: bold;
}
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
<div class="keepme remove remove2">
<div class="keepme2 remove remove3">x</div>
</div>
This avoids "selecting" non-matching elements, reducing the load, and of course no extra sorting is involved...

When to use NodeIterator

Benchmark compares QSA & .forEach vs a NodeIterator
toArray(document.querySelectorAll("div > a.klass")).forEach(function (node) {
// do something with node
});
var filter = {
acceptNode: function (node) {
var condition = node.parentNode.tagName === "DIV" &&
node.classList.contains("klass") &&
node.tagName === "A";
return condition ? NodeFilter.FILTER_ACCEPT : NodeFilter.FILTER_REJECT
}
}
// FIREFOX Y U SUCK
var iter = document.createNodeIterator(document, NodeFilter.SHOW_ELEMENT, filter, false);
var node;
while (node = iter.nextNode()) {
// do thing with node
}
Now either NodeIterator's suck or I'm doing it wrong.
Question: When should I use a NodeIterator ?
In case you don't know, DOM4 specifies what NodeIterator is.

NodeIterator (and TreeWalker, for that matter) are almost never used, because of a variety of reasons. This means that information on the topic is scarce and answers like #gsnedders' come to be, which completely miss the mark. I know this question is almost a decade old, so excuse my necromancy.
Initiation & Performance
=
It is true that the initiation of a NodeIterator is waaay slower than a method like querySelectorAll, but that is not the performance you should be measuring.
The thing about NodeIterators is that they are live-ish in the way that, just like an HTMLCollection or live NodeList, you can keep using the object after initiating it once.
The NodeList returned by querySelectorAll is static and will have to be re-initiated every time you need to match newly added elements.
This version of the jsPerf puts the NodeIterator in the preparation code. The actual test only tries to loop over all newly added elements with iter.nextNode(). You can see that the iterator is now orders of magnitudes faster.
Selector performance
=
Okay, cool. Caching the iterator is faster. This version, however, shows another significant difference. I've added 10 classes (done[0-9]) that the selectors shouldn't be matching. The iterator loses about 10% of its speed, while the querySelectors lose 20%.
On the other hand, this version, shows what happens when you add another div > at the start of the selector. The iterator loses 33% of its speed, while the querySelectors got a speed INCREASE of 10%.
Removing the initial div > at the start of the selector like in this version shows that both methods become slower, because they match more than earlier versions. Like expected, the iterator is relatively more performant than the querySelectors in this case.
This means that filtering on basis of a node's own properties (its classes, attributes, etc.) is probably faster in a NodeIterator, while having a lot of combinators (>, +, ~, etc.) in your selector probably means querySelectorAll is faster.This is especially true for the  (space) combinator. Selecting elements with querySelectorAll('article a') is way easier than manually looping over all parents of every a element, looking for one that has a tagName of 'ARTICLE'.
P.S. in §3.2, I give an example of how the exact opposite can be true if you want the opposite of what the space combinator does (exclude a tags with an article ancestor).
3 Impossible selectors
3.1 Simple hierarchical relationships
Of course, manually filtering elements gives you practically unlimited control. This means that you can filter out elements that would normally be impossible to match with CSS selectors. For example, CSS selectors can only "look back" in the way that selecting divs that are preceded by another div is possible with div + div. Selecting divs that are followed by another div is impossible.
However, inside a NodeFilter, you can achieve this by checking node.nextElementSibling.tagName === 'DIV'. The same goes for every selection CSS selectors can't make.
3.2 More global hierarchical relationships
Another thing I personally love about the usage of NodeFilters, is that when passed to a TreeWalker, you can reject a node and its whole sub-tree by returning NodeFilter.FILTER_REJECT instead of NodeFilter.FILTER_SKIP.
Imagine you want to iterate over all a tags on the page, except for ones with an article ancestor.With querySelectors, you'd type something like
let a = document.querySelectorAll('a')
a = Array.prototype.filter.call(a, function (node) {
while (node = node.parentElement) if (node.tagName === 'ARTICLE') return false
return true
})
While in a NodeFilter, you'd only have to type this
return node.tagName === 'ARTICLE' ? NodeFilter.FILTER_REJECT : // ✨ Magic happens here ✨
node.tagName === 'A' ? NodeFilter.FILTER_ACCEPT :
NodeFilter.FILTER_SKIP
In conclusion
You don't initiate the API every time you need to iterate over nodes of the same kind. Sadly, that assumption was made with the question being asked, and the +500 answer (giving it a lot more credit) doesn't even address the error or any of the perks NodeIterators have.
There's two main advantages NodeIterators have to offer:
Live-ishness, as discussed in §1
Advanced filtering, as discussed in §3
(I can't stress enough how useful the NodeFilter.FILTER_REJECT example is)
However, don't use NodeIterators when any of the following is true:
Its instance is only going to be used once/a few times
Complex hierarchical relationships are queried that are possible with CSS selectors
(i.e. body.no-js article > div > div a[href^="/"])
Sorry for the long answer :)

It's slow for a variety of reasons. Most obviously is the fact that nobody uses it so quite simply far less time has been spent optimizing it. The other problem is it's massively re-entrant, every node having to call into JS and run the filter function.
If you look at revision three of the benchmark, you'll find I've added a reimplementation of what the iterator is doing using getElementsByTagName("*") and then running an identical filter on that. As the results show, it's massively quicker. Going JS -> C++ -> JS is slow.
Filtering the nodes entirely in JS (the getElementsByTagName case) or C++ (the querySelectorAll case) is far quicker than doing it by repeatedly crossing the boundary.
Note also selector matching, as used by querySelectorAll, is comparatively smart: it does right-to-left matching and is based on pre-computed caches (most browsers will iterate over a cached list of all elements with the class "klass", check if it's an a element, and then check if the parent is a div) and hence they won't even bother with iterating over the entire document.
Given that, when to use NodeIterator? Basically never in JavaScript, at least. In languages such as Java (undoubtedly the primary reason why there's an interface called NodeIterator), it will likely be just as quick as anything else, as then your filter will be in the same language as the filter. Apart from that, the only other time it makes sense is in languages where the memory usage of creating a Node object is far greater than the internal representation of the Node.

Whats happening? One day its OK, the next day its 'undefined'?

I am writing a greasemonkey script. Recently i had this same problem twice and i have no idea why is this happening.
function colli(){
.....
var oPriorityMass = bynID('massadderPriority');//my own document.getElementById() function
var aPriorities = [];
if (oPriorityMass) {
for (var cEntry=0; cEntry < oPriorityMass.childNodes.length; cEntry++) {
var sCollNumber = oPriorityMass.childNodes[cEntry].getAttribute('coll');
if (bynID('adder' + sCollNumber + '_check').checked)
aPriorities.push(parseInt(sCollNumber));
}
}
.....
}
So the mystery of this is, one day i had oPriorityMass named as oPririoty. It was working fine, but the whole function was not yet complete and i started working on another functions for my script. These functions have no connection with each other.
Few days later i decided to go back to my function in the above example and finish it. I ran a test on it without modifying anything and got an error in the firefox's (4) javascript error console saying that oPriority.chilNodes[cEntry] is undefined. NOTE, few days back i have tested it exactly the same way and there was no such problem at all.
Ok, so, i decided to rename oPriority to oPriorityMass. Magically, problem got solved.
At first i thought, maybe there was some conflict of 2 objects, with the same name being used in different functions, which somehow continued to live even outside of function scope. My script is currently over 6000 lines big, but i did a search and found out that oPriority was not mentioned anywhere else but in this exact function.
Can somebody tell me, how and why is this happening? I mentioned same thing happened twice now and they happened in different functions, but the same problem node.childNodes[c] is undefined yet node is not null and node.childNodes.length show correct child count.
What is going on? How do i avoid such problems?
Thank you
EDIT: The error given by error console is
Error: uncaught exception: TypeError: oPriorityMass.childNodes[cEntry] is undefined
In response to Brocks comment:
GM_log(oPriorityMass.childNodes[cEntry]) returns undefined as a message. So node.childNodes[c] is the thing that is undefined in general.
My script creates a div window. Later, the above function uses elements in this div. Elements do have unique IDs and i am 100% sure the original site don't know about them.
My script has a start/stop button to run one or the other function when i need to.
I have been refreshing the page and running my script function now. I have noticed that sometimes (but not always) script will fail with the described error on the first run, however, if i run it again (without refreshing the page) it starts working.
The page has a javascript that modifies it. It changes some of it's element widths so it changes when the browser is resized. But i know it has no effect on my div as it is left unchanged when i resize browser.
EDIT2:
function bynID(sID) {
return top.document.getElementById(ns(sID));
}
function ns(sText) {
return g_sScriptName + '_' + sText;
}
ns function just adds the script name in front of the ID. I use it when creating HTML element so my elements never have the same id as the web page. So bynID() is simple function that saves some typing time when i need to get element by ID.
I have modified my colli() function to include check
if (oPriorityMass) {
if (!oPriorityMass.childNodes[0]) {
GM_log('Retrying');
setTimeout(loadPage,2000);
return;
}
for (var cEntry=0; cEntry < oPriorityMass.childNodes.length; cEntry++) {
var sCollNumber = oPriorityMass.childNodes[cEntry].getAttribute('coll');
if (bynID('adder' + sCollNumber + '_check').checked)
aPriorities.push(parseInt(sCollNumber));
}
}
The loadPage function does 1 AJAX call, then i run few XPATH queries on it, but the actual contents are never appended/shown on the page, just kept inside document.createElement('div'), then this function calls colli(). So now, as i have modified my function, i checked the error console and saw that it may take up to 5 tries for it to start working correctly. 5 x 2seconds, thats 10 seconds. It is never 5 retries always, may vary There's got to be something else going on?

In Firefox, childNodes can include #text nodes. You should check to make sure that childNodes[cEntry] has nodeType == 1 or has a getAttribute method before trying to call it. e.g.
<div id="d0">
</div>
<div id="d1"></div>
In the above in Firefox and similar browsers (i.e. based on Gecko and WebKit based browsers like Safari), d0 has one child node, a text node, and d1 has no child nodes.
So I would do something like:
var sCollNumber, el0, el1;
if (oPriorityMass) {
for (var cEntry=0; cEntry < oPriorityMass.childNodes.length; cEntry++) {
el0 = oPriorityMass.childNodes[cEntry];
// Make sure have an HTMLElement that will
// have a getAttribute method
if (el0.nodeType == 1) {
sCollNumber = el0.getAttribute('coll');
el1 = bynID('adder' + sCollNumber + '_check');
// Make sure el1 is not falsey before attempting to
// access properties
if (el1 && el1.checked)
// Never call parseInt on strings without a radix
// Or use some other method to convert to Number
aPriorities.push(parseInt(sCollNumber, 10));
}
}
Given that sCollNumber seems like it is a string integer (just guessing but it seems likely), you can also use:
Number(sCollNumber)
or
+sCollNumber
whichever suits and is more maintainable.

So, according to your last edit, it now works, with the delay, right?
But when I suggested the delay it was not meant to do (even more?) ajax calls while waiting!!
NOT:
if (!oPriorityMass.childNodes[0]) {
GM_log('Retrying');
setTimeout(loadPage,2000);
return;
More like:
setTimeout (colli, 2000);
So the ajax and the other stuff that loadPage does could explain the excessive delay.
The random behavior could be caused by:
return top.document.getElementById(ns(sID));
This will cause erratic behavior if any frames or iframes are present, and you do not block operation on frames. (If you do block such operation then top is redundant and unnecessary.)
GM does not operate correctly in such cases -- depending on what the script does -- often seeming to "switch" from top scope to frame scope or vice versa.
So, it's probably best to change that to:
return document.getElementById (ns (sID) );
And make sure you have:
if (window.top != window.self) //-- Don't run on frames or iframes
return;
as the top lines of code.
Beyond that, it's near impossible to see the problem, because of insufficient information.
Either boil the problem into a Complete, Self Contained, Recipe for duplicating the failure.
OR, post or link to the Complete, Unedited, Script.

JS 1-2k variables make page load slow

On one page of my website the user has the ability to choose and remove up to 2000 items through selecting multiple string representations of them in a dropdown list.
On page load, the objects are loaded onto the page from a previous session into 7 different drop-down lists.
In the window.onload event, the function looping through the items in the drop-downs makes an internal collection of the objects by adding them to a global array - This makes the page ridiculously slow to load, so, I'm fairly certain probably doing it wrong!
How else am I supposed to store these variables?
This is their internal representation:
function Permission(PName, DCID, ID) {
this.PName = PName;
this.DCID = DCID;
this.ID = ID;
}
where: PName is string. DCID is int. ID is int.
EDIT:
Thanks for the quick replies! I appreciate the help, I'm not great with JS! Here is more information:
'selectChangeEvent' is added to the Change and Click event of the Drop down list.
function selectChangeEvent(e) {
//...
addListItem(id);
//...
}
'addListItem(id)' sets up the visual representation of the objects and then calls :
function addListObject(x, idOfCaller) {
var arIDOfCaller = idOfCaller.toString().split('-');
if (arIDOfCaller[0] == "selLocs") {
var loc = new AccessLocation(x, arIDOfCaller[1]);
arrayLocations[GlobalIndexLocations] = loc;
GlobalIndexLocations++;
totalLocations++;
}
else {
var perm = new Permission(x, arIDOfCaller[1], arIDOfCaller[2]);
arrayPermissions[GlobalIndexPermissions] = perm;
GlobalIndexPermissions++;
totalPermissions++;
}
}

Still not enough to go on, but there are some small improvements I can see.
Instead of this pattern:
var loc = new AccessLocation(x, arIDOfCaller[1]);
arrayLocations[GlobalIndexLocations] = loc;
GlobalIndexLocations++;
totalLocations++;
which seems to involve redundant counters and has surplus assignment operations, try:
arrayLocations[arrayLocations.length] = new AccessLocation(x, arIDOfCaller[1]);
and just use arrayLocations.length where you would refer to GlobalIndexLocations or totalLocations (which fromt he code above would seem to always be the same value).
That should gain you a little boost, but this is not your main problem. I suggest you add some debugging Date objects to work out where the bottleneck is.

You may want to consider a design change to support the load. Some sort of paged result set or similar, to cut down on the number of concurrent records being modified.
As much as we desperately want them to be, browsers aren't quite there yet in terms of script execution speed that allow us to do certain types of heavy lifting on the client.

While I haven't tested this idea, I figured I'd throw it out there - might it be faster to return a JSON string from the server side, where your array is fully calculated on that side?
From that point, I'd wager that eval()'ing it (as evil as this may be) might be fast enough to where you could then write the contents onto the page, and your array setup would already be taken care of.
Then again, I suppose the amount of work it'd take the browser to construct the 2k new objects and inject them into the DOM wouldn't necessarily help the speed side of things in the end. At the end of the day, a design change is probably necessary, but sometimes we're stuck with what we've got, eh?