We Keep Coding

Javascript for loop does not wait for fetch request to complete and moves onto next iteration

I have below code in javascript in which some asynchronous task is being performed:
async function fetchData(id){
for(let i=1;;++i){
res = await fetch(`https://some-api/v1/products/${id}/data?page=${i}`,{//headers here});
res = await res.json();
if(res.length==0) break;
else{ //do some work here and continue for next iteration}
}
}
async function callApi(){
var arr = [//list of id's here to pass to api one by one, almost 100 id's here];
await Promise.all(arr.map(async(e)=>{
await fetchData(e);
}));
}
callApi();
The above code looks fine to me, except that it doesn't work as expected. Ideally, what should happen is that unless one id's call is not completed( unless break condition not satisfies for one id), the for loop should not proceed to next iteration. Rather, I am getting totally different results. The api calls are happening in random order because the loop is not waiting the iteration to complete. My hard requirement is that unless one iteration is not complete, it should not move to next one.
await seems to have no effect here. Please guide me how can I achieve this. I am running out of ideas.
Thank You!

Your arr.map(...) is not awaiting the different fetchData calls before the next map call, so I'd turn this into a specific for loop to be sure it waits:
async function callApi(){
const arr = [...];
for(let i = 0; i < arr.length; i++){
await fetchData(arr[i]);
}
}
or alternatively use a for of
async function callApi(){
const arr = [...];
for(let a of arr){
await fetchData(a);
}
}
The fetchData function also looks like it could use some improvements with error handling, but since you shortened your code quite a bit, I'm assuming there is something like that going on there, too, and your issue is actually with the callApi() code instead, as the fetch and await looks good to me there.

You should decide either to use promises or async await. Don't mix them.
With promises you can always use funky abstractions but with a simple recursive approach you can do like
function fetchData(hds, id, page = 1, pages = []){
return fetch(`https://some-api/v1/products/${id}/data?page=${page}`,hds)
.then(r => r.ok ? r.json() : Promise.reject({status:r.status,pages})
.then(j => fetchData(hds, id, ++page, pages.push(doSomethingWith(j))))
.catch(e => (console.log(e.status), e.pages));
}
So we use recursion to fetch indefinitelly until the API says enough and r.ok is false.
At the callApi side you can use reduce since we have an ids array.
const ids = [/* ids array */],
hds = { /* headers object */ };
function callApi(ids){
return ids.reduce( (p,id) => p.then(_ => fetchData(hds,id))
.then(pages => process(pages))
, Promise.resolve(null)
)
.catch(e => console.log(e));
}
So now both accesses to the id and page data are working asynchronously but only fired once the previous one finishes. Such as
(id=1,page=1) then (id=1,page=2) then (id=1,page=3) then (process 3 pages of id=1) then
(id=2,page=1) then (id=2,page=2) then (process 2 pages of id=2) etc...
While I love the promises, you can also implement the same functionality with the asyc await abstraction. I believe the idea behind the invention of the async await is to mimic sync imperative code. But keep in mind that it's an abstraction over an abstraction and I urge you to learn promises by heart before even attemting to use async await. The general rule is to never mix both in the same code.
Accordingly the above code could have been written as follows by using async await.
async function fetchData(hds, id){
let page = 1,
pages = [],
res;
while(true){
res = await fetch(`https://some-api/v1/products/${id}/data?page=${page++}`,hds);
if (res.ok) pages.push(await res.json())
else return pages;
}
}
Then the callApi function can be implemented in a similar fashion
const ids = [/* ids array */],
hds = { /* headers object */ };
async function callApi(ids){
let pages;
for(let i = 0; i < ids.length; i++){
try {
pages = await fetchData(hds,ids[i]);
await process(pages); // no need for await if the process function is sync
}
catch(e){
console.log(e);
}
}
}

how to call multiple await in a for loop if the one await depends on another awaits data in nodejs

Currently I am trying to call multiple awaits within a for loop, as per the documentation this is a performance heavy, so I was thinking of using promise.all() but the problem I am facing is I need the first awaits data to call the other awaits values with in the for loop and assign the new values in an empty object that I created outside the for loop. Do you know how to use promise.all() to solve this problem?
This is my current code:
const parsedSchema = {}
const arrayOfValues = Object.keys(objectOfValues);
for (let i = 0; i < arrayOfValues.length; i++) {
const arrayOfValuesSchema = (
await getObjectFromExternalAPI(arrayOfValues[i])
).data;
Object.assign(parsedSchema, {
...(await $RefParser.dereference(JSON.parse(arrayOfValuesSchema.toString('utf-8')))).properties
});
}
Update:
https://jsfiddle.net/sy4j6mgu/ this worked for me but I don't know how to simplify from here.

Does this help?
The following will synchronously and rapid-fire create promises for each value in arrayOfValues. Each executor function runs synchronously and tees-up async requests to getObjectFromExternalAPI and then $RefParser.dereference on the result.
The code will wait for all the promises to be fulfilled. Control then moves to synchronous aggregation of the results into a single object using Array#reduce.
const arrayOfValues = Object.keys(objectOfValues)
const promises = arrayOfValues.map((v) => new Promise((resolve) =>
getObjectFromExternalAPI(v)
.then(({ data }) => $RefParser.dereference(data.toString('utf-8')))
.then(resolve)
const results = await Promise.all(promises) // fails fast!
const parsedSchema = results.reduce((acc, { properties }) =>
({ ...acc, ...properties }), {})

Javascript: await inside of loop issue

I want to use Eslint plugin in my project with Webpack but it does not let me use await inside the loop.
According to Eslint docs it recommends to remove await from the loop and just add a Promise after.
Wrong example:
async function foo(things) {
const results = [];
for (const thing of things) {
// Bad: each loop iteration is delayed until the entire asynchronous operation completes
results.push(await bar(thing));
}
return baz(results);
}
Correct example:
async function foo(things) {
const results = [];
for (const thing of things) {
// Good: all asynchronous operations are immediately started.
results.push(bar(thing));
}
// Now that all the asynchronous operations are running, here we wait until they all complete.
return baz(await Promise.all(results));
}
But in my code I just merge data into one array which comes from HTTP request:
async update() {
let array = [];
for (url of this.myUrls) {
const response = await this.getData(url);
array = await array.concat(response);
}
}
Is it possible to remove await from this loop and add Promise just for array concatenation? I don't have an idea how to do it...

If you like one-liners.
const array = await Promise.all(this.myUrls.map((url)=> this.getData(url)));
In this case, the map method returns a bunch of promises, based on the URL, and your getData method. The Promise.all waits until all of your promises will be resolved. These promises run parallel.

You can use promise like this:
function update() {
let array = [],req=[];
for (url of this.myUrls) {
req.push(this.getData(url));
}
return Promise.all(req).then((data)=>{
console.log(data);
return data;
})
}

If I'm understanding you correctly, your getData function returns an array?
Using the one-liner supplied by Anarno, we can wait until all promises are resolved and then concatenate all the arrays.
const allResults = await Promise.all(this.myUrls.map((url) => this.getData(url)));
let finalArray = [];
allResults.forEach((item) => finalArray = finalArray.concat(item));

How can I make the promises to stop after getting x results in Firestore

I have this code that is checking if my userContacts ids exist in another collection, and I'm returning all the matches.
async function fetchCommonNumbers() {
var commonNumbers = [];
let contactsReference = admin.firestore().collection("user_contacts").doc("iNaYVsDCg3PWsDu67h75xZ9v2vh1").collection("contacts");
const dbContactReference = admin.firestore().collection('db_contacts_meta');
userContacts = await contactsReference.get();
userContacts = userContacts.docs;
await Promise.all(
userContacts.map(userContact => {
const DocumentID = userContact.ref.id;
//Check if Document exists
return dbContactReference.doc(DocumentID).get().then(dbContact => {
if (dbContact.exists) {
console.log(DocumentID);
commonNumbers.push(dbContact.data());
}
});
}));
return Promise.resolve(commonNumbers);
}
I need to only return X matches and not all since later I'll be having million of records and I want to reduce processing time.
How can I make the Promise.all to stop when commonNumbers has X items in it?

Currently there is not implementation of cancelable promises (more info can be found here enter link description here),
If you want, you can define your own "cancelable promise" wrapping a normal promise.

Reduce the processing time without "stopping" promises
You can't really make the promises stop. But since you're looking to reduce the number of database calls, what you can do is to selectively resolve your promises.
For example you can include a conditional statement in your map function. Like this
if commonNumbers.length < maxLength then return me a Promise containing the database call
Else, just resolve a random value (like false in my example)
Your promises will still be there, but you will have limited the number of DB calls to the necessary. It will look something like this
const arr = [1, 2, 3, 4, 5, 6];
const buffer = [];
const maxLenBuffer = 3;
const p = Promise.all(
arr.map(n => {
if (buffer.length < maxLenBuffer) {
buffer.push(n);
return Promise.resolve(n);
} else {
// There's still a promise to be resolved, but it's not a HTTP call
// This gives you the gain of performance you're looking for
return Promise.resolve(false);
}
})
);
p.then(() => console.log(buffer));
Note
While this can reduce your database calls, the actual number of calls can be a little higher than your maximum specified. This is due to the asynchronous nature of the calls

Instead of breaking the promise in between, I would suggest you use the limit method do firestore.
You can query only for X number of records and this X can be either hardcoded or can come from user. Something like:
documentRef.orderBy("name").limit(3).get()

Data not accessible outside Promise.all()

I have the below code to access movie sessions from a cinema site. I am looping using a while loop to fetch movie sessions.
And I intend to add the sessions within the loop to array sessionResults which is declared outside the while loop.
R. refers to the Ramda library
let page // passed as an argument to the outer function
let sessionResults = [];
while (currentCinemaIndex < cinemaList.length) {
await page.goto("www.fakeurl.com");
const _movies = await movies({ page });
//Get the sessions for each #_movies
const _movieSessions = await _movies.map(
async (_movie, index) => {
//sessions() returns an array of objects
const res = (await sessions({ page: page }, index + 1)).map(session => {
return Object.assign({}, _movie, session);
});
return res;
},
{ page }
);
//!!! AREA OF CONCERN
console.log(_movieSessions); // array of promises
Promise.all(_movieSessions).then(p => {
sessionResults = R.concat(R.flatten(p), sessionResults);
// console.log(sessionResults); // concatenated array
});
console.log(sessionResults); // []
//while loop logic
currentCinemaIndex = //increment currentCinemaIndex
limit =// set new limit
If you look at //!!! AREA OF CONCERN I have documented the value of sessionResults at different places.
Could you please advise why the value of sessionResults is not carried through outside Promise.all()?

You don't get the updated value of sessionResults because by the time the code executes until the console.log(sessionResults) after Promise.all(...), the promise has not resolved yet.
Therefore, the sessionResults returned by console.log is not yet updated.
What you can do instead is use await like below:
p = await Promise.all(_movieSesions);
sessionResults = R.concat(R.flatten(p), sessionResults);
console.log(sessionResults);
Please note that if you would use await like above, you need to do it inside an async function scope and not the global scope (because it is not async).

await Promise.all() worked based on comment from #CertainPerformance
Revised code looks like
sessionResults = await Promise.all(_movieSessions).then(p => R.flatten(p));
console.log(sessionResults); // concatenated array