I'm trying to make some checks before saving an array of objects (objects[]) to the DB (mongoDB using mongoose):
Those objects are already sorted by date, so objects[0].date is lower than objects[1].date.
Each object should check that last related saved object has a different value (to avoid saving the same info two times). This means that I've to query to the DB before each save, to make that check, AND each of these object MUST be stored in order to be able to make the check with the right object. If objects are not stored in order, the last related saved object might not be the correct one.
In-depth explanation:
HTTP request is send to an API. It returns an array of objects (sortered by date) that I want to process and save on my Mongo DB (using mongoose). I've to iterate through all these objects and, for each:
Look for the previous related object stored on DB (which COULD BE one of that array).
Check some values between the 'already stored' and the object to save to evaluate if new object must be saved or could be discarded.
Save it or discard it, and then jump to next iteration.
It's important to wait each iteration to finish because:
Items on array MUST be stored in DB in order: first those which lower date, because each could be modified by some object stored later with a higher date.
If next iteration starts before previous has finished, the query that searchs for the previous object could not find it if it hasn't been stored yet
Already tried:
Using promises or async/await on forEach/for loops only makes that iteration async, but it keeps launching all iterations at once.
I've tried using async/await functions inside forEach/for loops, even creating my own asyncForEach function as shown below, but none of this has worked:
Array.prototype.asyncForEach = function(fn) {
return this.reduce(
(promise, n) => promise.then(() => fn(n)),
Promise.resolve()
);
};
Test function:
let testArray = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
testArray.asyncForEach(function(element) {
setTimeout(() => {
console.log(element);
}, Math.random() * 500);
});
Provided example should show numbers on order in every case. It's not a problem if internal function (setTimeout in the example) should return a promise.
What I think I need is a loop that waits some function/promise between iterations, and only starts the next iteration when the first is already finished.
How could I do that? Thanks you in advance!
const myArray = ['a','b','c','d'];
async function wait(ms) { // comment 3
return new Promise(resolve => setTimeout(resolve, ms));
}
async function doSomething() {
await myArray.reduce(async (promise, item) => {
await promise; // comment 2
await wait(1000);
// here we could await something else that is async like DB call
document.getElementById('results').append(`${item} `);
}, Promise.resolve()); // comment 1
}
setTimeout(() => doSomething(), 1000);
<div id="results">Starting in 1 second <br/></div>
You can also use reduce and async await which you already said you've tried.
Basically, if you read how reduce works you can see that it accepts 2 parameters, first being callback to execute over each step and second optional initial value.
In the callback we have first argument being an accumulator which means that it accepts whatever the previous step returns or the optional initial value for first step.
1) You are giving initial value of promise resolve so that you start your first step.
2) Because of this await promise you will never go into next step until previous one has finished, since that is the accumulator value from previous step, which is promise since we said that callback is async. We are not resolving promise per say here, but as soon as the previous step is finish, we are going to implicitly resolve it and go to next step.
3) You can put for example await wait(30) to be sure that you are throttling the Ajax requests and not sending to many requests to 3rd party API's, since then there is no way that you will send more than 1000/30 requests per second, even if your code executes really fast on your machine.
Hm, ok i am not 100% sure if i understand your question in the right way. But if you try to perform an async array operation that awaits for your logic for each item, you can do it like follow:
async loadAllUsers() {
const test = [1,2,3,4];
const users = [];
for (const index in test) {
// make some magic or transform data or something else
await users.push(test[index])
}
return users;
}
Then you can simply invoke this function with "await". I hope that helps you.
In asyncForEach function you are resolving a Promise, setTimeout doesn't return a Promise.So if you convert your setTimeout to Promise. It will work as expected.
Here is the modified code:
testArray.asyncForEach(function(element) {
return new Promise((resolve, reject) => {
setTimeout(() => {
console.log(element);
return resolve(element)
}, Math.random() * 500);
})
});
Related
Trouble finding the reason why JavaScript for loop is not executing. Wrote 2 simple functions below that I want to execute and run i.e.: Bought method should try to "simulate" synchronous code.
The problem is that for some reason the for loop in the addNodes() method is never executed. However, if I run this separately i.e. for example line by line
var result = [];
var addressBookNodes = await generateAddressBooksNodes();
addNodes(result, addressBookNodes);
that tells me that the code is running fine however most likely it has something to do with the asynchronous nature of the generateAddressBooksNodes method. If I simply run the command :
var addressBookNodes = await generateAddressBooksNodes();
in the browser, I get an array of objects exactly what I was expecting to get. Basically the generateAddressBooksNodes returns a promise and when that promise is resolved I can see the correct array returned however what I do not understand why the for loop is not executed if the nodes objects have at least one element as shown in the picture below.
function addNodes(result, nodes){
console.log("3");
console.log(nodes);
for (var num in nodes) {
console.log("4");
let singleNode = nodes[num];
console.log(singleNode);
console.log("5");
result.push(singleNode);
}
}
async function getAddressBookAndContactNodes() {
var result = [];
console.log("1");
var addressBookNodesPromise = generateAddressBooksNodes();
addressBookNodesPromise.then( (arrayOfAddressBookNodes) => {
console.log("2");
addNodes(result, arrayOfAddressBookNodes);
})
return result;
}
Update 26 August 2020 :
After poking around the "arrayOfAddressBookNodes" object i noticed something really strange. I added additional print statement and printed the length of the "arrayOfAddressBookNodes" array. The length of the array is 0 when runned in the function. I do not understand how the length can be 0 if the object is printed shortly before the for loop and as shown on the picture below the length there is :1. What the hell is going on here?
I found another article i.e. JavaScript Array.length returning 0 that is basically explaining this. And in one of the commends it has been mentioned to use Map instead of an Array. I decided to use Set, and still get the same error i.e. the size of the set is 0 although the Set contains an object. i.e. below is the code and the picture of that execution.
async function getAddressBookAndContactNodes() {
var result = new Set();
console.log("1");
var addressBookNodes = await generateAddressBooksNodes();
console.log("2");
console.log(addressBookNodes);
console.log("3");
console.log("addressBookNodes size : " + addressBookNodes.size);
addressBookNodes.forEach(function(value) {
result.add(value);
});
console.log("6");
console.log(result);
console.log("7");
return result;
}
example using set
all this is really confusing to someone having a c++ backgroud, it makes my head explode.
Update 2 : 26 August 2020.
Ok i solved my problem. The problem was that the the promises are not working withing the for loop everything is explained here.
i need to use the regular "for (index = 0; index < contactsArray.length; ++index) " instead of foreach. after that it all worked. Somehow this leaves the impression that the tools of the language are broken in so many ways.
If generateAddressBooksNodes is returning a promise, you can use async to wait for the results:
async function getAddressBookAndContactNodes() {
var result = [];
console.log("1");
var addressBookNodesPromise = await generateAddressBooksNodes();
// note await here. Also, unless you're using this at a later time in your code, you can save space by not assigning it to a variable and simply returning generateAddressBooksNodes
addNodes(result, arrayOfAddressBookNodes);
return result;
}
I realized when I do something like this:
for (const entity of someArr) {
console.log('start now!')
await doSomeAsycAction()
console.log('waited X secs!')
}
It prints out:
start now!
waited X secs!
start now!
waited X secs!
...
But when I use map:
arr.map(async entity => {
console.log('start now!')
await doSomeAsycAction()
console.log('waited X secs!')
})
It prints out:
start now!
start now!
start now!
...
waited X secs!
waited X secs!
waited X secs!
...
Can someone explain why this is the case?
The difference between the two flows is that the first one (using for, for..in or for..of) is running the loop iterations sequentially and the other one (using map, filter, reduce, forEach and so on) are running (in case async symbol is used somewhere in the mapping function) concurrently (kind of).
In for loops, the next iteration must wait for the previous one to finish. This allows you to do some async operations relevant for the next iteration.
In contrast, using the async methods runs each iteration independently, so you can't rely on other iterations in your current iteration. Those kind of functions receive a function as an argument and executes it immediately for every item in the array.
Think of every iteration as an independent promise execution. When running an async function, the await symbol tells that this operation might take a while (i.e I/O, DB calls, network operations...) and let's the code outside of the current executed function keep going (and resume later on, after the async call returns). The map function sees that the current iteration is busy and go on to the next one. Somewhen in the future it would resume and execute console.log('waited X secs!').
You can simulate the same behavior of async executions with for loop this way (would maybe help demonstrating the difference):
for (const entity of someArr) {
(async () => {
console.log('start now!')
await doSomeAsycAction()
console.log('waited X secs!')
})()
}
The async-await syntax is working per function scope, and map is defining a new function scope (the function passed as the param to the function) just like the (anonymous) function that gets executed each iteration in my example. Wish it helps to understand.
One important thing to notice is that each iteration of the map doesn't return the mapped value you were expecting for, but a promise that will be resolved with this value. So if you try to rely on one of the mapped array values - you must add an await right before it, otherwise the value type would still be a promise. Take a look at the following example:
let arr = [1];
arr = arr.map(async entity => incrementAsync(entity));
console.log(arr[0]) // would print an unresolved Promise object
console.log(await arr[0]) // would print 2
Basic array prototype loop functions like forEach, map, filter, find etc. don't wait for next iteration.Their basic behavior is to iterate not awaiting. If you want use like waiting function then try to use like below
for (const event of events) {
if (failure or conditional) {
continue;
}
}
This question already has an answer here:
How to chain and share prior results with Promises [duplicate]
(1 answer)
Closed 6 years ago.
Is there a better way to do this?
let foo;
return functionA().then(result => {
foo = result;
return functionB();
}).then(bar => {
return functionC(foo, bar);
});
Notice that the result of functionA is required input to functionC. Using a variable outside the promise scope works fine, but it feels kinda icky. Is there a clean idiomatic way to do this?
Please note that I do not have the opportunity to change the API of any of the functions I am calling.
You could try using Promise.all() which you can pass an array of promises and it provides an array of responses within the then() callback when all promises passed in have resolved. You can access those array values to pass into functionC:
Promise.all([functionA, functionB]).then(values => functionC(values[0], values[1]));
Might be a little cleaner (without nesting) as it doesn't look like the response from functionA needs to be passed into functionB.
Otherwise, nesting would look like:
return functionA().then(foo => {
return functionB().then(bar => {
return functionC(foo, bar);
});
});
One option is, as Alexander Staroselsky writes, to use Promise.all(functionA(), functionB()). This runs the two functions simultaneously. If that's what you want to happen, you can use that answer. If, however, you want them to happen one after the other and then also be able to pass the result onto another handler, you can do this:
function functionA() {
return new Promise(resolve => resolve(1));
}
function functionB() {
return new Promise(resolve => resolve(2));
}
function functionC(first, second) {
return first + second;
}
functionA()
.then(result => Promise.all([result, functionB()]))
.then(function([first, second]) {
return functionC(first, second);
})
.then(result => {
console.log(result);
});
The functions are obviously simplified -- but the lovely thing about Promises is that that doesn't matter: we don't care how complex they are or how long they take. Yay Promises!
The clever thing is that Promise.all doesn't mind if the values you pass are not Promises. If they are any other value, they are treated as a Promise that is resolved immediately. (In the same way that you can do Promise.resolve(42).then(...).) So we can do Promise.all([result, functionB()]). This says "give me a Promise that is resolved when you have a final value for both result and functionB() and pass both values on". That is immediately in the case of result and at some unspecified time in the case of functionB.
The returned values are then passed as an array to the next then function.
.then(function([first, second]) {
return functionC(first, second);
})
This then receives the values as an array (see the use of destructuring in the parameter list) and sends those values on to functionC. We then do one last then function to show the result.
I constantly run into problems with this pattern with callbacks inside loops:
while(input.notEnd()) {
input.next();
checkInput(input, (success) => {
if (success) {
console.log(`Input ${input.combo} works!`);
}
});
}
The goal here is to check every possible value of input, and display the ones that pass an asynchronous test after confirmed. Assume the checkInput function performs this test, returning a boolean pass/fail, and is part of an external library and can't be modified.
Let's say input cycles through all combinations of a multi-code electronic jewelry safe, with .next incrementing the combination, .combo reading out the current combination, and checkInput asynchronously checking if the combination is correct. The correct combinations are 05-30-96, 18-22-09, 59-62-53, 68-82-01 are 85-55-85. What you'd expect to see as output is something like this:
Input 05-30-96 works!
Input 18-22-09 works!
Input 59-62-53 works!
Input 68-82-01 works!
Input 85-55-85 works!
Instead, because by the time the callback is called, input has already advanced an indeterminate amount of times, and the loop has likely already terminated, you're likely to see something like the following:
Input 99-99-99 works!
Input 99-99-99 works!
Input 99-99-99 works!
Input 99-99-99 works!
Input 99-99-99 works!
If the loop has terminated, at least it will be obvious something is wrong. If the checkInput function is particularly fast, or the loop particularly slow, you might get random outputs depending on where input happens to be at the moment the callback checks it.
This is a ridiculously difficult bug to track down if you find your output is completely random, and the hint for me tends to be that you always get the expected number of outputs, they're just wrong.
This is usually when I make up some convoluted solution to try to preserve or pass along the inputs, which works if there is a small number of them, but really doesn't when you have billions of inputs, of which a very small number are successful (hint, hint, combination locks are actually a great example here).
Is there a general purpose solution here, to pass the values into the callback as they were when the function with the callback first evaluated them?
If you want to iterate one async operation at a time, you cannot use a while loop. Asynchronous operations in Javascript are NOT blocking. So, what your while loop does is run through the entire loop calling checkInput() on every value and then, at some future time, each of the callbacks get called. They may not even get called in the desired order.
So, you have two options here depending upon how you want it to work.
First, you could use a different kind of loop that only advances to the next iteration of the loop when the async operation completes.
Or, second, you could run them all in a parallel like you were doing and capture the state of your object uniquely for each callback.
I'm assuming that what you probably want to do is to sequence your async operations (first option).
Sequencing async operations
Here's how you could do that (works in either ES5 or ES6):
function next() {
if (input.notEnd()) {
input.next();
checkInput(input, success => {
if (success) {
// because we are still on the current iteration of the loop
// the value of input is still valid
console.log(`Input ${input.combo} works!`);
}
// do next iteration
next();
});
}
}
next();
Run in parallel, save relevant properties in local scope in ES6
If you wanted to run them all in parallel like your original code was doing, but still be able to reference the right input.combo property in the callback, then you'd have to save that property in a closure (2nd option above) which let makes fairly easy because it is separately block scoped for each iteration of your while loop and thus retains its value for when the callback runs and is not overwritten by other iterations of the loop (requires ES6 support for let):
while(input.notEnd()) {
input.next();
// let makes a block scoped variable that will be separate for each
// iteration of the loop
let combo = input.combo;
checkInput(input, (success) => {
if (success) {
console.log(`Input ${combo} works!`);
}
});
}
Run in parallel, save relevant properties in local scope in ES5
In ES5, you could introduce a function scope to solve the same problem that let does in ES6 (make a new scope for each iteration of the loop):
while(input.notEnd()) {
input.next();
// create function scope to save value separately for each
// iteration of the loop
(function() {
var combo = input.combo;
checkInput(input, (success) => {
if (success) {
console.log(`Input ${combo} works!`);
}
});
})();
}
You could use the new feature async await for asynchronous calls, this would let you wait for the checkInput method to finish when inside the loop.
You can read more about async await here
I believe the snippet below achieves what you are after, I created a MockInput function that should mock the behaviour of your input. Note the Async and await keywords in the doAsyncThing method and keep an eye on the console when running it.
Hope this clarifies things.
function MockInput() {
this.currentIndex = 0;
this.list = ["05-30-96", "18-22-09", "59-62-53", "68-82-0", "85-55-85"];
this.notEnd = function(){
return this.currentIndex <= 4;
};
this.next = function(){
this.currentIndex++;
};
this.combo = function(){
return this.list[this.currentIndex];
}
}
function checkInput(input){
return new Promise(resolve => {
setTimeout(()=> {
var isValid = input.currentIndex % 2 > 0; // 'random' true or false
resolve( `Input ${input.currentIndex} - ${input.combo()} ${isValid ? 'works!' : 'did not work'}`);
}, 1000);
});
}
async function doAsyncThing(){
var input = new MockInput();
while(input.notEnd()) {
var result = await checkInput(input);
console.log(result);
input.next();
}
console.log('Done!');
}
doAsyncThing();
The following example code confuses me...
"use strict";
var filesToLoad = [ 'fileA','fileB','fileC' ];
var promiseArray = [];
for( let i in filesToLoad ) {
promiseArray.push(
new Promise( function(resolve, reject ) {
setTimeout( function() {
resolve( filesToLoad[i] );
}, Math.random() * 1000 );
})
);
}
Promise.all( promiseArray ).then(function(value) {
console.log(value);
});
The reason I'm confused is that I was expecting a random ordered output on the console. But I always get the following...
[ 'fileA', 'fileB', 'fileC' ]
That confuses me little to say the least, but what really gets me scratching my head is when I change the let i to var i I get the following result....
[ 'fileC', 'fileC', 'fileC' ]
As someone who has only recently tried to fully understand Promises and not that long ago starting using let, I'm really stumped.
Further reading...
After getting lots of great answers I have refactored the example to get rid of the loop and i. Will seem obvious to most, but fun for me...
"use strict";
var filesToLoad = [ 'fileA','fileB','fileC' ];
function start( name ) {
return new Promise( function(resolve, reject ) {
setTimeout( function() {
resolve( name + '_done' );
}, Math.random() * 1000 );
});
}
Promise.all( filesToLoad.map(start) ).then(function(value) {
console.log(value);
});
It is because of closure. Read about it here and here.
Also let is block scoped whereas var is function scoped.
In case of using var i:
After timeout when the function is triggered, looping was completed and i was set to 2. so it got resolved with filesToLoad[2] for all the setTimeout functions.
In case of using let i:
Since it is block scoped, when function is resolved it remembers the state of i when setTimeOut was declared, so when it gets resolved it uses correct value of i.
Regarding the order of output in case of using let i.
Promise.all(Iterable<any>|Promise<Iterable<any>> input) -> Promise
Given an Iterable(arrays are Iterable), or a promise of an Iterable, which produces promises (or a mix of promises and values), iterate over all the values in the Iterable into an array and return a promise that is fulfilled when all the items in the array are fulfilled. The promise's fulfillment value is an array with fulfillment values at respective positions to the original array. If any promise in the array rejects, the returned promise is rejected with the rejection reason.
So irrespective of order in which your promises gets resolved, the result of promise.all will always have promise resolve value in correct order.
Why using let and var produces different results:
The reason that using let produces the desired result over var is that when using let you declare a block-scoped variable, such that when the loop moves to another iteration the value of i remains unaffected for the contents of the loop at that time.
Defining a var variable in the header of a for-loop does not mean that it exists only for the life of the for-loop, as you will notice if you do the following:
for (var i = 0; i < 10; i++) { /*...*/ }
console.log(i); //=> 10
// `i` is already declared and its value will be 10
for (; i < 20; i++) { /*...*/ }
console.log(i); //=> 20
You can avoid this problem altogether if you use Array#forEach, which does the job of filesToLoad[i] for you by giving you the next value within a callback function on each iteration:
filesToLoad.forEach((file) => {
promiseArray.push(
new Promise( function(resolve, reject ) {
setTimeout( function() {
resolve( file );
}, Math.random() * 1000 );
})
);
});
______
Does using either let or var affect the behaviour of Promise#all?
No. In your example, the position of the Promises in promiseArray defines in what order the values are added to the results array, not when each of those Promises is resolved. The fact that you resolve the Promises at random intervals does not move the position of the resolved value within promiseArray. What you have demonstrated is that Promise#all produces an array of values whose positions are mapped to the Promise that produced their value.
See this answer for more information about the behaviour of Promise#all:
All this means that the output is strictly ordered as the input as long as the input is strictly ordered (for example, an array).
It's because var's scope isn't a child of for, it's a sibling. So the loop runs once and sets i = 0. It then runs another 2 more times and sets i = 1 and then i = 2. After this has all happened, the timeouts then run and all run the resolve function and passes in resolve( filesToLoad[2] ). let works correctly because the value of let i is not overridden by the following loop iterations.
In short, the timeout only runs after the loop has already run 3 times and therefore passes in the same value. I've created a jsfiddle of a working version using var.
These are actually two different, unrelated problems.
Your first observation, that the output is in the original order, rather than a random order, is due to the behaviour of Promise.all. Promise.all returns a new promise that resolves to an array when all sub-promises have resolved, in the original order. You can think of it like Array.map, but for promises.
As stated in the Bluebird (a JS promise library) documentation:
The promise's fulfillment value is an array with fulfillment values at respective positions to the original array.
Your second problem, how changing let to var changes the output to have all of the same values, is due to array scoping. Put simply, a variable defined with var exists outside of the for loop scope. This means that on each iteration, you are actually modifying the value of the pre-existing variable, as opposed to creating a new variable. Since your code inside the Promise executes later, the variable will have the value from the last loop iteration.
let creates a variable that exists inside that for iteration. This means that, later on in the Promise, you are accessing a variable defined in that iterations scope, which isn't overridden by the next loop iteration.
That's the difference between let and var. Your issue is not directly related to Promises (besides the fact that they run async).
TL;DR
var declarations are moved to beginning of function by interpreter, so when timeouts executed the loop has already run to end of array on the same variable. let keeps the variable in scope, so each iteration of the loop used a different variable.
Background:
Javascript has a feature called hoisting, which means that a variable or function is always declared at the beginning of it's wrapping function. If the coder doesn't, the interpreter moves it. Hence:
var i;
for (i in array)
....
and
for (var i in array)
....
are equivalent, even though, having some reason, or other programming background, you would expect i to be scoped to the loop alone in the second case.
That's why
alert(i);
...
var i=5;
alerts undefined instead of throwing an error. As far as the interpreter is concerned - there was a declaration. It's like
var i;
alert(i);
...
i=5;
Now:
ES6/ES2015 has introduced let - which behaves as our first assumption in the second example above.
alert(i);
...
let i=5;
will actually throw an error. When reaching the alert there is no i to speak of.
In your case
With let you went through the loop with a new variable each time, since let was defined only to scope of the loop. Each iteration is a scope, each used a different variable (though all named i in their respective scopes)
When using var, you actually declared i before the loop
var i;
for( i in filesToLoad ) {
promiseArray.push(
new Promise( function(resolve, reject ) {
setTimeout( function() {
resolve( filesToLoad[i] );
}, Math.random() * 1000 );
})
);
}
so it's the same variable on each iteration (or, in each scope), meaning that the loop assigned the last item to i before any of the timeouts returned. That's why the result was 3 times the last item.