I'm currently iterating through an array, making an api request, waiting for it to resolve, then moving onto the next api call.
I've read of using Promise.all() and I think I could make these api calls in parallel and wait for that promise to resolve, but I'm not exactly sure how to translate what I have here to use Promise.all().
async lockFolder(folderModel) {
const driveId = folderModel.driveId;
// THIS IS WHAT I'D LIKE TO TRANSLATE TO USE Promise.all()
for (const file of folderModel.docs) {
let res = await this._lockFile(file.id, driveId);
}
// return something....
}
async _lockFile(fileId, driveId) {
try {
return await axios.post(myRequestOmittedForBrevity);
} catch (err) {
//some error
}
}
Is there a good way to translate my loop to Promise.all()? Can I still use await for it's response? Most examples I've seen use .then() but I've been trying to stick with await. Any help is appreciated!
It makes total sense to use Promise.all or Promise.allSettled for this, as soon your requests aren't dependent on each other.
It can be something like this:
async lockFolder(folderModel) {
const driveId = folderModel.driveId;
// THIS IS WHAT I'D LIKE TO TRANSLATE TO USE Promise.all()
const listOfPendingPromises = folderModel.docs.map(file => this._lockFile(file.id, driveId))
const resultArray = await Promise.all(listOfPendingPromises)
// return something....
}
async _lockFile(fileId, driveId) {
return await axios.post(myRequestOmittedForBrevity);
}
Related
I have below code in javascript in which some asynchronous task is being performed:
async function fetchData(id){
for(let i=1;;++i){
res = await fetch(`https://some-api/v1/products/${id}/data?page=${i}`,{//headers here});
res = await res.json();
if(res.length==0) break;
else{ //do some work here and continue for next iteration}
}
}
async function callApi(){
var arr = [//list of id's here to pass to api one by one, almost 100 id's here];
await Promise.all(arr.map(async(e)=>{
await fetchData(e);
}));
}
callApi();
The above code looks fine to me, except that it doesn't work as expected. Ideally, what should happen is that unless one id's call is not completed( unless break condition not satisfies for one id), the for loop should not proceed to next iteration. Rather, I am getting totally different results. The api calls are happening in random order because the loop is not waiting the iteration to complete. My hard requirement is that unless one iteration is not complete, it should not move to next one.
await seems to have no effect here. Please guide me how can I achieve this. I am running out of ideas.
Thank You!
Your arr.map(...) is not awaiting the different fetchData calls before the next map call, so I'd turn this into a specific for loop to be sure it waits:
async function callApi(){
const arr = [...];
for(let i = 0; i < arr.length; i++){
await fetchData(arr[i]);
}
}
or alternatively use a for of
async function callApi(){
const arr = [...];
for(let a of arr){
await fetchData(a);
}
}
The fetchData function also looks like it could use some improvements with error handling, but since you shortened your code quite a bit, I'm assuming there is something like that going on there, too, and your issue is actually with the callApi() code instead, as the fetch and await looks good to me there.
You should decide either to use promises or async await. Don't mix them.
With promises you can always use funky abstractions but with a simple recursive approach you can do like
function fetchData(hds, id, page = 1, pages = []){
return fetch(`https://some-api/v1/products/${id}/data?page=${page}`,hds)
.then(r => r.ok ? r.json() : Promise.reject({status:r.status,pages})
.then(j => fetchData(hds, id, ++page, pages.push(doSomethingWith(j))))
.catch(e => (console.log(e.status), e.pages));
}
So we use recursion to fetch indefinitelly until the API says enough and r.ok is false.
At the callApi side you can use reduce since we have an ids array.
const ids = [/* ids array */],
hds = { /* headers object */ };
function callApi(ids){
return ids.reduce( (p,id) => p.then(_ => fetchData(hds,id))
.then(pages => process(pages))
, Promise.resolve(null)
)
.catch(e => console.log(e));
}
So now both accesses to the id and page data are working asynchronously but only fired once the previous one finishes. Such as
(id=1,page=1) then (id=1,page=2) then (id=1,page=3) then (process 3 pages of id=1) then
(id=2,page=1) then (id=2,page=2) then (process 2 pages of id=2) etc...
While I love the promises, you can also implement the same functionality with the asyc await abstraction. I believe the idea behind the invention of the async await is to mimic sync imperative code. But keep in mind that it's an abstraction over an abstraction and I urge you to learn promises by heart before even attemting to use async await. The general rule is to never mix both in the same code.
Accordingly the above code could have been written as follows by using async await.
async function fetchData(hds, id){
let page = 1,
pages = [],
res;
while(true){
res = await fetch(`https://some-api/v1/products/${id}/data?page=${page++}`,hds);
if (res.ok) pages.push(await res.json())
else return pages;
}
}
Then the callApi function can be implemented in a similar fashion
const ids = [/* ids array */],
hds = { /* headers object */ };
async function callApi(ids){
let pages;
for(let i = 0; i < ids.length; i++){
try {
pages = await fetchData(hds,ids[i]);
await process(pages); // no need for await if the process function is sync
}
catch(e){
console.log(e);
}
}
}
What I am trying to do:
I am trying to retrieve all comments and their replies from firestore (firebase database) using a recursive method. Here is the structure of the data:
What is the problem
The parent asynchronous function does not wait for the nested asynchronous function to complete.
getThread = async (req, res) => {
// Getting comments belonging to thread
const thread_document = await db.doc(`/Threads/${req.params.threadid}`).get()
threadData = thread_document.data()
threadData.threadid = thread_document.id
const comment_query = await db.collection('Comments').where('threadid', '==', threadData.threadid).get()
// Getting replies belonging to comments
for (document of comment_query){
let commentData = await getReplies(document.id)
threadData.comments.push(commentData )
}
return res.json(threadData)
}
//Recursive function to retrieve replies
getReplies = async (id) => {
let comment = await db.doc(`/Comments/${id}`).get()
let commentData = comment.data()
commentData.comment_replies = commentData.replies.map(idx => {
// The parent async function does not wait for the the async function here to finish.
// Placing a await keyword here will raise the error 'await is only valid in async functions and the top level bodies of modules'
return getReplies(idx)
})
console.log(commentData)
return commentData
}
Given the following example,
Since the parent async function does not wait for the nested async function, the order of execution now is A -> B -> a, and a fails to be mapped into commentData and commentData for comment A would end up empty. Hence, I want to program to do A -> a -> B. To do so, I would like to place a await keyword just before the getReplies like
return await getReplies(idx)
but it will raise the error,
await is only valid in async functions and the top level bodies of modules.
Which is confusing as getReplies is already a async function. I've looked into other solutions in stackoverflow but I am not able to get the recursive function working. Any insights would be appreciated, thank you.
commentData.comment_replies = commentData.replies.map(idx => {
// ...
return getReplies(idx)
})
This map statement is going to create an array of promises, but not wait for those promises to finish. You should use Promise.all to combine them into a single promise, and then await that promise to get the array of comment replies:
const promises = commentData.replies.map(idx => {
return getReplies(idx);
});
commentData.comment_replies = await Promise.all(promises);
Which is confusing as getReplies is already a async function.
You were getting that error because the function you're inside is idx => { return getReplies(idx) }, which is not an async function. But putting an await in there is not a solution to your problem anyway.
I want to use Eslint plugin in my project with Webpack but it does not let me use await inside the loop.
According to Eslint docs it recommends to remove await from the loop and just add a Promise after.
Wrong example:
async function foo(things) {
const results = [];
for (const thing of things) {
// Bad: each loop iteration is delayed until the entire asynchronous operation completes
results.push(await bar(thing));
}
return baz(results);
}
Correct example:
async function foo(things) {
const results = [];
for (const thing of things) {
// Good: all asynchronous operations are immediately started.
results.push(bar(thing));
}
// Now that all the asynchronous operations are running, here we wait until they all complete.
return baz(await Promise.all(results));
}
But in my code I just merge data into one array which comes from HTTP request:
async update() {
let array = [];
for (url of this.myUrls) {
const response = await this.getData(url);
array = await array.concat(response);
}
}
Is it possible to remove await from this loop and add Promise just for array concatenation? I don't have an idea how to do it...
If you like one-liners.
const array = await Promise.all(this.myUrls.map((url)=> this.getData(url)));
In this case, the map method returns a bunch of promises, based on the URL, and your getData method. The Promise.all waits until all of your promises will be resolved. These promises run parallel.
You can use promise like this:
function update() {
let array = [],req=[];
for (url of this.myUrls) {
req.push(this.getData(url));
}
return Promise.all(req).then((data)=>{
console.log(data);
return data;
})
}
If I'm understanding you correctly, your getData function returns an array?
Using the one-liner supplied by Anarno, we can wait until all promises are resolved and then concatenate all the arrays.
const allResults = await Promise.all(this.myUrls.map((url) => this.getData(url)));
let finalArray = [];
allResults.forEach((item) => finalArray = finalArray.concat(item));
I have the below async function taking in an array of 4 objects. The first one is processed in a .map loop that executes a query and results in an error (in the await executeQuery(queryString)). When this first object is completed processing, the loop ends and the other 3 objects are not processed.
async function myFunction(arrayOfObjects, param1) {
const promises = arrayOfObjects.map(async function (currElement, i) {
var queryString = 'www.querystring.com?APIparameter=param1';
const executeQuery = util.promisify(request);
await executeQuery(queryString).then(data => {
//process data here using currElement, i
}).catch(err => console.log('error: ', err));
});
await Promise.all(promises).catch();
}
I originally only had await Promise.all(promises) and thought revising it to await Promise.all(promises).catch(); would do the trick but it is still failing out on the first object iterated.
I was wondering how best to achieve continuing the .map loop after an error hits the catch in the executeQuery.
Please and thank you!
Not 100% sure what's happening with this code but there are a few things I would change. Firstly it's not good practice to use an await inside a loop and secondly, your promises array is probably not what you expect as the await is yielding until each promise is resolved inside the loop. Return the Promise from the map function to generate an array of Promises that will each resolve and process the data in their own time. You can then await all the promises and catch any errors using try/catch. Something similar to the following should do what you are expecting...
async function myFunction(arrayOfObjects, param1) {
const executeQuery = util.promisify(request);
const promises = arrayOfObjects.map((currElement, i) => {
const queryString = 'www.querystring.com?APIparameter=param1';
return executeQuery(queryString).then(data => {
// process data here using currElement, i
});
});
try {
await Promise.all(promises);
// all promises will be resolved here
} catch (err) {
console.log('error: ', err);
}
}
I have a function with multiple forEach loops:
async insertKpbDocument(jsonFile) {
jsonFile.doc.annotations.forEach((annotation) => {
annotation.entities.forEach(async (entity) => {
await this.addVertex(entity);
});
annotation.relations.forEach(async (relation) => {
await this.addRelation(relation);
});
});
return jsonFile;
}
I need to make sure that the async code in the forEach loop calling the this.addVertex function is really done before executing the next one.
But when I log variables, It seems that the this.addRelation function is called before the first loop is really over.
So I tried adding await terms before every loops like so :
await jsonFile.doc.annotations.forEach(async (annotation) => {
await annotation.entities.forEach(async (entity) => {
await this.addVertex(entity);
});
await annotation.relations.forEach(async (relation) => {
await this.addRelation(relation);
});
});
But same behavior.
Maybe it is the log function that have a latency? Any ideas?
As we've discussed, await does not pause a .forEach() loop and does not make the 2nd item of the iteration wait for the first item to be processed. So, if you're really trying to do asynchronous sequencing of items, you can't really accomplish it with a .forEach() loop.
For this type of problem, async/await works really well with a plain for loop because they do pause the execution of the actual for statement to give you sequencing of asynchronous operations which it appears is what you want. Plus, it even works with nested for loops because they are all in the same function scope:
To show you how much simpler this can be using for/of and await, it could be done like this:
async insertKpbDocument(jsonFile) {
for (let annotation of jsonFile.doc.annotations) {
for (let entity of annotation.entities) {
await this.addVertex(entity);
}
for (let relation of annotation.relations) {
await this.addRelation(relation);
}
}
return jsonFile;
}
You get to write synchronous-like code that is actually sequencing asynchronous operations.
If you are really avoiding any for loop, and your real requirement is only that all calls to addVertex() come before any calls to addRelation(), then you can do this where you use .map() instead of .forEach() and you collect an array of promises that you then use Promise.all() to wait on the whole array of promises:
insertKpbDocument(jsonFile) {
return Promise.all(jsonFile.doc.annotations.map(async annotation => {
await Promise.all(annotation.entities.map(entity => this.addVertex(entity)));
await Promise.all(annotation.relations.map(relation => this.addRelation(relation)));
})).then(() => jsonFile);
}
To fully understand how this works, this runs all addVertex() calls in parallel for one annotation, waits for them all to finish, then runs all the addRelation() calls in parallel for one annotation, then waits for them all to finish. It runs all the annotations themselves in parallel. So, this isn't very much actual sequencing except within an annotation, but you accepted an answer that has this same sequencing and said it works so I show a little simpler version of this for completeness.
If you really need to sequence each individual addVertex() call so you don't call the next one until the previous one is done and you're still not going to use a for loop, then you can use the .reduce() promise pattern put into a helper function to manually sequence asynchronous access to an array:
// helper function to sequence asynchronous iteration of an array
// fn returns a promise and is passed an array item as an argument
function sequence(array, fn) {
return array.reduce((p, item) => {
return p.then(() => {
return fn(item);
});
}, Promise.resolve());
}
insertKpbDocument(jsonFile) {
return sequence(jsonFile.doc.annotations, async (annotation) => {
await sequence(annotation.entities, entity => this.addVertex(entity));
await sequence(annotation.relations, relation => this.addRelation(relation));
}).then(() => jsonFile);
}
This will completely sequence everything. It will do this type of order:
addVertex(annotation1)
addRelation(relation1);
addVertex(annotation2)
addRelation(relation2);
....
addVertex(annotationN);
addRelation(relationN);
where it waits for each operation to finish before going onto the next one.
foreach will return void so awaiting it will not do much. You can use map to return all the promises you create now in the forEach, and use Promise.all to await all:
async insertKpbDocument(jsonFile: { doc: { annotations: Array<{ entities: Array<{}>, relations: Array<{}> }> } }) {
await Promise.all(jsonFile.doc.annotations.map(async(annotation) => {
await Promise.all(annotation.entities.map(async (entity) => {
await this.addVertex(entity);
}));
await Promise.all(annotation.relations.map(async (relation) => {
await this.addRelation(relation);
}));
}));
return jsonFile;
}
I understand you can run all the addVertex concurrently. Combining reduce with map splitted into two different set of promises you can do it. My idea:
const first = jsonFile.doc.annotations.reduce((acc, annotation) => {
acc = acc.concat(annotation.entities.map(this.addVertex));
return acc;
}, []);
await Promise.all(first);
const second = jsonFile.doc.annotations.reduce((acc, annotation) => {
acc = acc.concat(annotation.relations.map(this.addRelation));
return acc;
}, []);
await Promise.all(second);
You have more loops, but it does what you need I think
forEach executes the callback against each element in the array and does not wait for anything. Using await is basically sugar for writing promise.then() and nesting everything that follows in the then() callback. But forEach doesn't return a promise, so await arr.forEach() is meaningless. The only reason it isn't a compile error is because the async/await spec says you can await anything, and if it isn't a promise you just get its value... forEach just gives you void.
If you want something to happen in sequence you can await in a for loop:
for (let i = 0; i < jsonFile.doc.annotations.length; i++) {
const annotation = jsonFile.doc.annotations[i];
for (let j = 0; j < annotation.entities.length; j++) {
const entity = annotation.entities[j];
await this.addVertex(entity);
});
// code here executes after all vertix have been added in order
Edit: While typing this a couple other answers and comments happened... you don't want to use a for loop, you can use Promise.all but there's still maybe some confusion, so I'll leave the above explanation in case it helps.
async/await does not within forEach.
A simple solution: Replace .forEach() with for(.. of ..) instead.
Details in this similar question.
If no-iterator linting rule is enabled, you will get a linting warning/error for using for(.. of ..). There are lots of discussion/opinions on this topic.
IMHO, this is a scenario where we can suppress the warning with eslint-disable-next-line or for the method/class.
Example:
const insertKpbDocument = async (jsonFile) => {
// eslint-disable-next-line no-iterator
for (let entity of annotation.entities) {
await this.addVertex(entity)
}
// eslint-disable-next-line no-iterator
for (let relation of annotation.relations) {
await this.addRelation(relation)
}
return jsonFile
}
The code is very readable and works as expected. To get similar functionality with .forEach(), we need some promises/observables acrobatics that i think is a waste of effort.