We Keep Coding

Uncaught TypeError while scraping data from bank

I aim to scrape my personal account information automatically using JavaScript. Below is the code I inserted into my browser console after logging in.
function wait(ms){
var start = new Date().getTime();
var end = start;
while(end < start + ms) {
end = new Date().getTime();
}
}
function click_1 () {
document.querySelector("#accountMaskLink-640332982").shadowRoot.querySelector("a > span.link__text").click();
}
function click_2 () {
document.querySelector("#downloadActivityIcon").shadowRoot.querySelector("button").click();
}
click_1();
wait(8000);
click_2();
Please notice that the wait function is to wait for the site to render the next available menu. However, this doesn't work. Despite 8 seconds are long enough for the next menu to show, it still gives the following error.
Uncaught TypeError: Cannot read property 'shadowRoot' of null
at <anonymous>:3:52
(anonymous) # VM3475:3
Question 1: This doesn't happen if I enter the code line by line. What's going on?
EDIT
Answer 1: This is provided by #Bravo in the comment. When I use the sync version of wait, the menu renderer stops working too. So I have to use the async version.

Thanks to #Bravo's hint, and some research, I've come up with a satisfactory answer.
Synchronous wait halts everything, including the menu renderer.
We need an async version.
function clickAccountInfo (ms) {
document.querySelector("#accountMaskLink-640332982").shadowRoot.querySelector("a > span.link__text").click();
return new Promise(resolve => setTimeout(resolve, ms));
}
function clickDownload (ms) {
document.querySelector("#downloadActivityIcon").shadowRoot.querySelector("button").click();
return new Promise(resolve => setTimeout(resolve, ms));
}
function selectMenuDownload () {
document.querySelector("#downloadActivityOptionId").shadowRoot.querySelector("#select-downloadActivityOptionId").click();
document.querySelector("#downloadActivityOptionId > mds-select-option:nth-child(2)").shadowRoot.querySelector("div").click();
document.querySelector("#download").shadowRoot.querySelector("button").click()
}
await clickAccountInfo(15000);
await clickDownload(5000);
selectMenuDownload();

Why is my code executing although it should pause?

I have an API which is limited regarding how many requests per minute (50/minute) I can send to any endpoint provided by that API.
In the following code-section, I filter the objects orders with an URL as property, every object with an URL that provides data should be stored in successfullResponses in my app.component.ts.
Promise.all(
orders.map(order => this.api.getURL(order.resource_url).catch(() => null))
).then(responses => {
const successfulResponses = responses.filter(response => response != null)
for(let data of successfulResponses) {
// some other requests should be sent with data here
}
});
There are more than 50 orders to check, but I just can check maximum 50 orders at once, so I try to handle it in my service. I set the first date when the first request is sent. After that I compare the dates of the new request with the first one. If the difference is over 60, I set the current date to the new one and set maxReq again to 50. If it is under 60, I check if there are requests left, if yes I send the request and if not I just wait one minute :
sleep(ms){
return new Promise(resolve => setTimeout(resolve, ms));
}
async getURL(){
if(!this.date){
let date = new Date();
this.date = date;
}
if((new Date().getSeconds() - this.date.getSeconds() > 60 )){
this.maxReq = 50;
this.date = new Date();
return this.http.get(url, this.httpOptions).toPromise();
} else {
if(this.maxReq > 0){
this.maxReq -= 1;
return this.http.get(url, this.httpOptions).toPromise();
} else{
console.log("wait");
await this.sleep(60*1000);
this.maxReq = 50;
this.date = new Date();
return this.http.get(url, this.httpOptions).toPromise();
}
}
}
However the code in app.component.tsis not waiting for the function getURL() and executes further code with requests which leads to the problem that I send ´too many requests too quickly´.
What can I do about that?

I had a similar problem while trying to use promises with multiple async functions. It's an easy thing to forget, but in order to make them all wait, you have to use await on the root line that calls the function in question.
I'm not entirely certain, but my presumption is that your await this.sleep(60*1000); line is indeed waiting for a timeout to occur, but whilst it is doing this, the code that called getURL() is executing the rest of its lines, because it did not have an await (or equivalent, like .then) before getURL().
The way I discovered this in my case was by using a good debugging tool (I used Chrome DevTools's own debugging features). I advise you do the same, adding breakpoints everywhere, and see where your code is going with each line.
Here is a short, rough example to show what I mean:
// This code increments a number from 1 to 2 to 3 and returns it each time after a delay of 1 second.
async function loop() {
for (i = 1; i <= 3; i++) {
console.log('Input start');
/* The following waits for result of aSync before continuing.
Without 'await', it would execute the last line
of this function whilst aSync's own 'await'
waited for its result.
--- This is where I think your code goes wrong. --- */
await aSync(i);
console.log('Input end');
}
}
async function aSync(num) {
console.log('Return start');
/* The following waits for the 1-second delay before continuing.
Without 'await', it would return a pending promise immediately
each time. */
let result = await new Promise(
// I'm not using arrow functions to show what it's doing more clearly.
function(rs, rj) {
setTimeout(
function() {
/* For those who didn't know, the following passes the number
into the 'resolved' ('rs') parameter of the promise's executor
function. Without doing this, the promise would never be fulfilled. */
rs(num);
}, 1000
)
}
);
console.log(result);
console.log('Return end');
}
loop();

How do you run a setTimeout once a second inside a forEach loop?

My code in order to create multiple placements in an online service that has a 60 write constraint per minute:
placementsToAdd.forEach((placement, index) => {
setTimeout(() => {
options.url = `https://api.company.com/placement?publisher_id=${existingPub ? existingPub : placementsJson[0].PublisherId}&site_id=${placement.siteId}`
options.body = `{"placement":{"name":"${placement.placement}"}}`
request(options, callback);
},1000 * (index + 1))
})
It works this way but I am concerned about the wait time if there are a list of placements of 2000 or 3000 at one time, the wait time might be excessively long.
Is there a better way to refactor this code in order to get my requests built one per second no matter what? Without that "* (index + 1)," it seems to keep trying to build all at once hitting the wall after 60.
I've tried to use promises and async await (which is new to me) but it doesn't seem to change the behavior.
Thanks!
As requested, show how I've tried to use promises with this code:
async function createThePlacements() {
let promise = new Promise((resolve, reject) => {
for (let i = 0; i < placementsToAdd.length; i++) {
setTimeout(() => {
options.url = `https://api.company.com/placement?publisher_id=${existingPub ? existingPub : placementsJson[0].PublisherId}&site_id=${placementsToAdd[i].siteId}`
options.body = `{"placement":{"name":"${placementsToAdd[i].placement}"}}`
request(options, callback);
},1000)
}
});
let result = await promise; // pause till the promise resolves
console.log('result - ', result);
}
createThePlacements();
So, bit of a disclaimer - as mentioned, I've never used Async Await before so reading up to try to understand how it works as well. This seems to be the syntax but my result doesn't seem to be anything at the moment but the code also continues to do what it's supposed to do, just trying to make all the calls in my test of 300 all at once.
Also, of note, i have a resolve inside the callback of the request call. It resolves so even the next parts of my app finish up all the way to the end. That's why I don't have a reject or resolve here.

How do you run a setTimeout once a second inside a forEach loop?
The most straightforward approach would be:
const wait = ms => new Promise(resolve => setTimeout(resolve, ms));
for (const placement of placementsToAdd) {
const options = {...};
request(options, callback);
await wait(1000);
}
await works predictably inside plain for-loops, not inside forEach.
I haven't touched your callback but it would need to handle errors. More refactoring is possible.
The most significant improvement here, I think, is that we're not pushing requests ahead of time. This way we retain control, and should needs change or anything go haywire, we can break out of the loop without spamming the server for another minute.

The best option would be to have a request method that returns a Promise.
Then you could rewrite your code like this.
function sleep(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
}
function requestPlacement(placement) {
const options = {...};
return request(options);
}
async function requestAllPlacements(placements) {
for(let i = 0; i < placements.length; i+=60) {
if (i > 0) {
// wait 1 minute
await(sleep(60000));
}
await Promise.all(
placements
.slice(i, 60)
.map(requestPlacement);
);
}
}

Node.js - how to call a Facebook API within a for loop without exceeding rate limit?

I have an array with almost 2 millions of Facebook idSender, and I want to itearate over it, calling a Facebook API for each one of them.
Now, due to the asynchronicity, if I launch a simple for loop, Facebook would return me a rate limit exceed error after 4-5 minutes.
After some attempts I found out that the ideal delay to avoid rate limit is 20 milliseconds.
So I tried this solution: I wrapped my function in a Promise, and I used setTimeout to set a delay.
async function asyncLoop(values) {
var prm = await new Promise(function(resolve, reject) {
setTimeout(function() {
for (var i=0; i<values.length; i++) {
check(values[i].idSender, values[i].iscrizione, values[i].id_sito)
if(checked == values.length) {
resolve()
}
}
},20);
});
return prm;
}
asyncLoop(idSenderArray)
but this solution it's not working, and I'm also aware it's almost never a good idea using setTimeout fuction to manipulate asynchronicity in Javascript, in fact I can't even say what exactly it's happening there.
Any suggestion for possible modifies to my function? Any idea on a totally new implementation that could work?
Thanks

Just await a time inside the for loop:
const timer = ms => new Promise(res => setTimeout(res, ms));
async function asyncLoop(values) {
for (var i = 0; i < values.length; i++) {
await timer(20);
check(values[i].idSender, values[i].iscrizione, values[i].id_sito)
}
}

You may also do something like this with promises;
var apiCall = (url,n) => new Promise((v,x) => setTimeout(v,50,`the result of API call # ${url} no ${n}`)),
calls = Array(20).fill()
.map((_,i) => new Promise((v,x) => setTimeout(v,20*i))
.then(_ => apiCall("http://facebook.com",i))
.then(v => console.log(`I am doing something with ${v}`)));
.as-console-wrapper{
height: 100%;
max-height: 100% !important
}

Alternatively, using setTimeout with promises can also be used:
async function asyncLoop(values) {
let apiCallPromises = values.map((value, index)=> new Promise(resolve=>{
setTimeout(_=>{
check(value.idSender, value.iscrizione, value.id_sito)
resolve()
}, index*20)
}))
return Promise.all(apiCallPromises);
}
Fairly strait-forward, it maps each value to a check() call with a delay of 20n ms for each subsequent request.

Put a Delay in Javascript

I need to add a delay of about 100 miliseconds to my Javascript code but I don't want to use the setTimeout function of the window object and I don't want to use a busy loop. Does anyone have any suggestions?

Unfortunately, setTimeout() is the only reliable way (not the only way, but the only reliable way) to pause the execution of the script without blocking the UI.
It's not that hard to use actually, instead of writing this:
var x = 1;
// Place mysterious code that blocks the thread for 100 ms.
x = x * 3 + 2;
var y = x / 2;
you use setTimeout() to rewrite it this way:
var x = 1;
var y = null; // To keep under proper scope
setTimeout(function() {
x = x * 3 + 2;
y = x / 2;
}, 100);
I understand that using setTimeout() involves more thought than a desirable sleep() function, but unfortunately the later doesn't exist. Many workarounds are there to try to implement such functions. Some using busy loops:
function sleep(milliseconds) {
var start = new Date().getTime();
for (var i = 0; i < 1e7; i++) {
if ((new Date().getTime() - start) > milliseconds){
break;
}
}
}
others using an XMLHttpRequest tied with a server script that sleeps for a amount of time before returning a result.
Unfortunately, those are workarounds and are likely to cause other problems (such as freezing browsers). It is recommended to simply stick with the recommended way, which is setTimeout()).

If you're okay with ES2017, await is good:
const DEF_DELAY = 1000;
function sleep(ms) {
return new Promise(resolve => setTimeout(resolve, ms || DEF_DELAY));
}
await sleep(100);
Note that the await part needs to be in an async function:
//IIAFE (immediately invoked async function expression)
(async()=>{
//Do some stuff
await sleep(100);
//Do some more stuff
})()

I just had an issue where I needed to solve this properly.
Via Ajax, a script gets X (0-10) messages.
What I wanted to do:
Add one message to the DOM every 10 Seconds.
the code I ended up with:
$.each(messages, function(idx, el){
window.setTimeout(function(){
doSomething(el);
},Math.floor(idx+1)*10000);
});
Basically, think of the timeouts as a "timeline" of your script.
This is what we WANT to code:
DoSomething();
WaitAndDoNothing(5000);
DoSomethingOther();
WaitAndDoNothing(5000);
DoEvenMore();
This is HOW WE NEED TO TELL IT TO THE JAVASCRIPT:
At Runtime 0 : DoSomething();
At Runtime 5000 : DoSomethingOther();
At Runtime 10000: DoEvenMore();
Hope this helps.

This thread has a good discussion and a useful solution:
function pause( iMilliseconds )
{
var sDialogScript = 'window.setTimeout( function () { window.close(); }, ' + iMilliseconds + ');';
window.showModalDialog('javascript:document.writeln ("<script>' + sDialogScript + '<' + '/script>")');
}
Unfortunately it appears that this doesn't work in some versions of IE, but the thread has many other worthy proposals if that proves to be a problem for you.

Actually only setTimeout is fine for that job and normally you cannot set exact delays with non determined methods as busy loops.

Use a AJAX function which will call a php page synchronously and then in that page you can put the php usleep() function which will act as a delay.
function delay(t){
var xmlhttp;
if (window.XMLHttpRequest)
{// code for IE7+, Firefox, Chrome, Opera, Safari
xmlhttp=new XMLHttpRequest();
}
else
{// code for IE6, IE5
xmlhttp=new ActiveXObject("Microsoft.XMLHTTP");
}
xmlhttp.open("POST","http://www.hklabs.org/files/delay.php?time="+t,false);
//This will call the page named delay.php and the response will be sent to a division with ID as "response"
xmlhttp.send();
document.getElementById("response").innerHTML=xmlhttp.responseText;
}
http://www.hklabs.org/articles/put-delay-in-javascript