We Keep Coding

Continuing a javascript after using .click() on a button which adds new DOM elements

I am a lowly operations employee without authorization to change the programs and permissions on my machine, and I would like to automate some highly repetitive data entry. I know there are a lot of programs that can do that, however, for the sake of this discussion we'll assume that I'm not allowed to have any of them and I can only script through the debug F12 menu in Chrome. I also probably don't understand half of these words as well as I should.
I have to run test cases on a third-party vendor's highly dynamic website, and I've already successfully written javascript which adds texts to elements in the DOM and presses the "next" button.
The problem is, upon .click()ing the "next" button, it takes time for the page to update, and the update creates new elements which weren't in the DOM when the script was initialized. I need to find a way to delay the execution of the script until the DOM contains all the elements I need to update.
As a really, really crude proof of concept I wrote the pre-filler for each page as a function, and I serially called each function at the end of the previous function, using setTimeout(nextfunct, 10000) to let the page update before executing the next line. (I was going to refine that by trying to create some kind of object listener instead of an arbitrary 10 second delay, but I wasn't even able to get that far.) This approach creates two errors.
1) The script seems to be checking whether the elements are on the DOM before the end of the setTimeout(), so it still gives me an error. If nextfunct is defined as
document.getElementById("doesntexistyet").value = "Fill Me";
console.log("nextfunct ran");
I will get the error message stating there is no element with the id "doesntexistyet" immediately, not after a delay of 10 seconds. The element on the next page will not update.
2) The DOM updating interrupts my script. In the above code, the console output will not ever appear in my console. If I comment out the missing element, so the function only prints a comment, it will still not appear in my console. However, if I comment out the code and I switch the setTimeout to 1ms, "nextfunct ran" will appear in my console, until the page updates, at which time the console will be deleted.
Are there ways around this which I can implement using only vanilla JS and a browser? I'm sure there's a keyword I can search for where someone has discussed this before, but it seems like the vast majority of JS autofilling discussions are oriented towards people designing code to be integrated into a website,
Thanks

Properly Await Propagation of InnerHTML to Complete Before Executing Function [duplicate]

I hope I won't make a fool of myself but I'm trying to understand what is happening in those two lines of code:
document.body.innerHTML = 'something';
alert('something else');
What I am observing is that alert shows before HTML has been updated (or maybe it has but the page hasn't been refreshed/repainted/whatever)
Checkout this codepen to see what I mean.
Please note that even putting alert in setTimeout(..., 0) does not help. Looks like it takes more event loops for innerHTML to actually update page.
EDIT:
I forgot to mention I am using Chrome and did not check other browsers. Looks like it's only visible in Chrome. Nevertheless I am still interested why is that happening.

Setting innerHTML is synchronous, as are most changes you can make to the DOM. However, rendering the webpage is a different story.
(Remember, DOM stands for "Document Object Model". It's just a "model", a representation of data. What the user sees on their screen is a picture of how that model should look. So, changing the model doesn't instantaneously change the picture - it take some time to update.)
Running JavaScript and rendering the webpage actually happen separately. To put it simplistically, first all of the JavaScript on the page runs (from the event loop - check out this excellent video for more detail) and then after that the browser renders any changes to the webpage for the user to see. This is why "blocking" is such a big deal - running computationally intensive code prevents the browser from getting past the "run JS" step and into the "render the page" step, causing the page to freeze or stutter.
Chrome's pipeline looks like this:
As you can see, all of the JavaScript happens first. Then the page gets styled, laid out, painted, and composited - the "render". Not all of this pipeline will execute every frame. It depends on what page elements changed, if any, and how they need to be rerendered.
Note: alert() is also synchronous and executes during the JavaScript step, which is why the alert dialog appears before you see changes to the webpage.
You might now ask "Hold on, what exactly gets run in that 'JavaScript' step in the pipeline? Does all my code run 60 times per second?" The answer is "no", and it goes back to how the JS event loop works. JS code only runs if it's in the stack - from things like event listeners, timeouts, whatever. See previous video (really).
https://developers.google.com/web/fundamentals/performance/rendering/

Yes, it is synchronous, because this works (go ahead, type it in your console):
document.body.innerHTML = 'text';
alert(document.body.innerHTML);// you will see a 'text' alert
The reason you see the alert before you see the page changing is that the browser rendering takes more time and isn't as fast as your javascript executing line by line.

The innerHTML property actual does get updated synchronously, but the visual redraw that this change causes happens asynchronously.
The visual rendering the DOM is asynchronous in Chrome and will not happen until after the current JavaScript function stack has cleared and the browser is free to accept a new event. Other browsers might use separate threads to handle JavaScript code and browser rendering, or they might let some events get priority while an alert is halting the execution of another event.
You can see this in two ways:
If you add for(var i=0; i<1000000; i++) { } before your alert, you've given the browser plenty of time to do a redraw, but it hasn't, because the function stack has not cleared (add is still running).
If you delay your alert via an asynchronous setTimeout(function() { alert('random'); }, 1), the redraw process will get to go ahead of the function delayed by setTimeout.
This does not work if you use a timeout of 0, possibly because Chrome gives event-queue priority to 0 timeouts ahead of any other events (or at least ahead of redraw events).

jquery waits the for loop to complete before executing previous events

I have a function which is triggered on a click event. Inside the function first line is to show an overlay, and after that there is a for loop. I expect the function to show the overlay first and then continue with the for loop.
Instead overlay is being shown only after the for loop completes.
Here is the jsFiddle Link
$(document).on("click",function(){
$("h1").text("Clicked");
for(var i=0;i<100000;i++){
console.log(i);
}
})

view will not update in the same thread or in same flow of execution. it will use invalidation technique. which means view updates postpone for some time, thats way we can do bunch of update in a minimal effort.
javascript is single threaded fashion, so view update will wait until for loop finish.

Use setTimeout() to have a delay between the overlay and for loop
https://jsfiddle.net/b9m5spxu/

Here is a good article that explains this behavior (thks #subash for the hint): http://javascript.info/tutorial/events-and-timing-depth
JavaScript execution and rendering
In most browsers, rendering and JavaScript use single event queue. It
means that while JavaScript is running, no rendering occurs.
Check it on the demo below. When you press run, the browser may halt
for some time, because it changes div.style.backgroundColor from
A00000 to #FFFFFF.
In most browsers, you see nothing until the script finishes, or until
the browser pauses it with a message that ‘a script is running too
long’.
The exception is Opera.
<div style="width:200px;height:50px;background-color:#A00000"></div>
<input type="button" onclick="run()" value="run()">
<script> function run() { var div =
document.getElementsByTagName('div')[0] for(var
i=0xA00000;i<0xFFFFFF;i++) {
div.style.backgroundColor = '#'+i.toString(16) } } </script>
In Opera, you may notice div is redrawn. Not every change causes a
repaint, probably because of Opera internal scheduling. That’s because
event queues for rendering and JavaScript are different in this
browser.
In other browsers the repaint is postponed until the JavaScript
finishes.
Again, the implementation may be different, but generally the nodes
are marked as “dirty” (want to be recalculated and redrawn), and
repaint is queued. Or, the browser may just look for dirty nodes after
every script and process them.
Immediate reflow The browser contains many optimizations to speedup
rendering and painting. Generally, it tries to postpone them until the
script is finished, but some actions require nodes to be rerendered
immediately.
For example: elem.innerHTML = 'new content' alert(elem.offsetHeight)
// <-- rerenders elem to get offsetHeight In the case above, the
browser has to perform relayouting to get the height. But it doesn’t
have to repaint elem on the screen.
Sometimes other dependant nodes may get involved into calculations.
This process is called reflow and may consume lots of resources if
script causes it often.
Surely, there’s much more to talk about rendering. It will be covered
by a separate article [todo].

How to check if JavaScript file is yet to loaded or loading

I am doing GUI automation of my website via Selenium (RobotFramework). The problem I am facing is:
When my automation script clicks on some element (button,link etc) which is supposed to perform some action, it dose nothing. This happening randomly. When we test is manually, it works all the time. One observation is, with slower machine/environment, this tends to happen more.
I suspect this is happening either due to some corresponding JS is not loaded yet or if there is any such thing called "action binding" with each elemetnt, has not happened.
Some question
- Is there a way to find out if all the JS calls are over?
- In case action binding happens, has it already bound or not.
Please share if you have any other solution.

do you know what is last to load on the page? This should be very easy to find out via Developer Tools in your browser of choice.
You can then easily use a Wait Until Keyword (there are many variations) to wait until that last item appears as you expect, then continue with your test. Setting the timeout length and interval will help control the overhead of time/performance.
Wait Until Element Is Visible id=finalElement 10 finalElement did not appear on the screen before timeout period
http://robotframework.org/Selenium2Library/Selenium2Library.html - please see the documentation for further examples and options in terms of keywords

I don't fully understand JavaScript Threading

Before I dive into the question. Let me state that by Event Loop I am referring to http://en.wikipedia.org/wiki/Event_loop. This is something that browsers implement. For more information, read this: http://javascript.info/tutorial/further-javascript-features/events-and-timing-depth.
This question is hard and long, so, please try to bear with it! And I do appreciate all answers!
So. Now, as I understand it, in JavaScript there is a single main thread (in most browser environments, that is). So, code like:
for (var color = 0x000; color < 0xfff; color++) {
$('div').css('background-color', color.toString(16));
}
will produce an animation from black to white, but you won't see that because the rendering is done after the code has been processed (when the next tick happens -- the browser enters the Event Loop).
If you want to see the animation, you could do:
for (var color = 0x000; color < 0xfff; color++) {
setTimeout(function() {
$('div').css('background-color', color.toString(16));
}, 0);
}
The above example would produce a visible animation, because setTimeout pushes a new event to the browser Event Loop stack which will be processed after there is nothing running (it enters the Event Loop to see what to do next).
It seems that the browser in this case have 0xfff (4095) events pushed into the stack, where each of them are processed with a render process in between them. So, my first question (#1) is that when exactly does the rendering take place? Does it always take place in between the processing of two events in the Event Loop stack?
The second question is about the code in the javascript.info website link I gave you.
...
function func() {
timer = setTimeout(func, 0)
div.style.backgroundColor = '#'+i.toString(16)
if (i++ == 0xFFFFFF) stop()
}
timer = setTimeout(func, 0)
....
My question here is that will the browser push a new "rendering" event to the Event Loop stack every time it reaches the div.style. ... = ... line? But does it not first push an event due to the setTimeout-call? So, does the browser end up in a stack like:
setTimeout event
render event
Since the setTimeout call was processed before the div style change? If that's how the stack looks like, then I would assume the next time the browser enters the Event Loop it will process the setTimeout's callback and end up having:
rendering event
setTimeout event
rendering event
and continue with the rendering event that the earlier setTimeout call produced?

Q1: Not necessarily. Browsers to varying degrees implement optimizations. For example, they may wait to collect several style changes before triggering an expensive recalculation of the layout. So the answer is: depends on the specific browser.
Try this: http://taligarsiel.com/Projects/howbrowserswork1.htm#Render_tree_construction (the document is dated Oct 2009 - i.e. it is sufficiently up to date)
Q2: The rendering is not necessarily the same as the JS execution - that's two different engines. Ths JS engine is not responsible for the rendering, it just interfaces with the render engine. It seems to me the main message for your second question is this independence of the JS from the rendering engine. Remember, a browser (or a webpage) does not need Javascript, their main purpose is to render HTML based on CSS style rules. Javascript is just one way to manipulate the HTML (the DOM tree really) and the style rules.
Note that you can force rendering by reading a style definition - at this point the rendering engine has no choice but process any outstanding style changes, especially if it involves any position changes. That's why one should remove objects from the rendering tree (e.g. by setting display:none - visibility:hidden is NOT enough since the element's size is still considered for layout) before doing a lot of style changes or adding a lot of elements, e.g. when lots of rows are added one by one (a "for" loop) to a table.
Not part of the question at all - but since I just mentioned a difference between display:none and visibility:hidden, that's also a consideration when adding hidden position:absolute elements like dialogs. While there is no visible difference whether an absolutely positioned element is hidden from you using one or the other method, internally there IS a big difference: when hidden using visibility:hidden the element is part of the rendering tree, with display:none it is not. So, if one has such an element that needs to be toggled a lot one should use visibility:hidden, because when the "display" style is switched between "none" and e.g. "block" the browser has to render it first.

The article you mention only considers Javascript. A lot more happens in the browser; reflowing and repainting are/can be triggered by a lot more things; take a look at the following links for more info on this.
http://www.phpied.com/rendering-repaint-reflowrelayout-restyle/
http://www.browserscope.org/reflow/about
I wouldn't use setTimeout for this purpose.
Edit:
As per the comments, the recommended way is to use requestAnimationFrame. As of this writing, this is only available in unstable releases of most browsers. There are however several libraries available providing cross-browser access to it, and fall back to using setTimeout if necessary.
Take a look at this demo for an example working in old browsers, as well as in new ones:
http://paulirish.com/2011/requestanimationframe-for-smart-animating/