We Keep Coding

Node JS multliple promises chaining

I have node JS api server and I'm having issues with correct chaining of the Promises:
app.post(
"/api/tasks",
async function (_req, res) {
const newArray = [{ MyTasks: [] }];
const getOne = async (owner, taskID) => {
return await getOneDocument(owner, taskID).then((result) => {
console.log("get one doc", result);
return result;
});
};
// first promise
let toApproveTasks = await getToApproveTasks(_req.body.userID);
console.log("1", toApproveTasks);
// loop trough the result of 1st promise and run async function for each
const arrayToDoc = async (array) => {
array.TasksToApprove.forEach(async (element) => {
let objToPush = await getOne(element.Owner, element.TaskID);
console.log("1.5", objToPush);
newArray.MyTasks.push(objToPush);
});
};
// second promise
await arrayToDoc(toApproveTasks);
console.log("2", newArray);
// third promise
let finalResult = await parseCosmosOutput(newArray);
console.log("3", finalResult);
res.status(200).send(finalResult);
}
);
What I get in console is :
1 [Object] - all good
Emppty Array
Empty Array
get one doc {object} - all good
1.5 {object} - all good
How would I make sure when I loop over result of 1st promise my code awaits async function and pushes to newArray results ?

Use For..of instead of forEach inside arrayToDoc function
E.g
const arrayToDoc = async (array) => {
for(let element of array.TasksToApprove){
let objToPush = await getOne(element.Owner, element.TaskID);
console.log("1.5", objToPush);
newArray.MyTasks.push(objToPush);
}
};

How to push failed promises to separate array?

I am looping through ids that sends it to an async function and I want to return both success and failed data, right now I am only returning success data
const successContractSignature: LpContractSla[] = [];
for (const id of lpContractSlaIds) {
const data = await createLpSignature(context, id);
if (data) {
successContractSignature.push(data);
}
}
return successContractSignature;
If the createLpSignature throws error, do i need a try catch here? but wouldnt that exit the loop?
How can I push failed data to separate array without breaking the loop?

Unless there's a specific reason to avoid the call in parallel, it's always a good practice to avoid to start async calls in a for loop with await, since you are going to wait for each promise to resolve (or reject ) before executing the next one.
This is a better pattern which lets you also get all the results of the promises, either they resolved or rejected:
const successContractSignature: LpContractSla[] = await Promise.allSettled(lpContractSlaIds.map((id: string) => createLpSignature(context,id)))
return successContractSignature;
But if for some particular reason you need to make these calls in a sequence and not in parallel, you can wrap the single call in a try catch block ,that won't exit the loop:
for (const id of lpContractSlaIds) {
let data;
try {
data = await createLpSignature(context, id);
} catch(e) {
data = e
}
if (data) {
successContractSignature.push(data);
}
}
You can test it in this example:
const service = (id) =>
new Promise((res, rej) =>
setTimeout(
() => (id %2 === 0 ? res("ID: "+id) : rej('Error ID : '+id)),
1000
)
);
const ids = [1,2,3,4,5]
const testParallelService = async () => {
try {
const data = await Promise.allSettled(ids.map(id => service(id)))
return data.map(o => `${o.status}: ${o.reason ?? o.value}`)
} catch(e) {
console.log(e)
}
}
testParallelService().then(data => console.log("Parallel data: ", data))
const testSequentialService = async () => {
const res = [];
for (const id of ids) {
let data;
try {
data = await service(id);
} catch (e) {
data = e;
}
if (data) {
res.push(data);
}
}
return res;
};
testSequentialService().then((data) => console.log('Sequential Data: ', data));

using async await with reduce

i have following
const imageField = ["logoUrl", "fullLogoUrl"]
const onCreate = async (submitData: any) => {
const uploadImageField = await imageField.reduce(
async function (acc: any, cur: string) {
await acc;
const url = await uploadImage(submitData.general[cur][0]);
acc[cur] = url;
return acc;
},
{}
);
console.log(uploadImageField);
}
this is my console.log
{
logoUrl: "https://........"
}
only logoUrl field is show, fullLogoUrl is missing

The problem is that acc on the second iteration is a promise object - that's why you await it. However, you still assign the [cur] property on that promise object, not on the promise result, and the implicit promise chaining of the async function as well as the explicit awaits will just ignore properties on the promise object. You could fix this by doing acc = await acc;, but really I recommend not to use reduce with async/await at all. A normal loop is much simpler and has no pitfalls.
const imageField = ["logoUrl", "fullLogoUrl"]
const onCreate = async (submitData: any) => {
const uploadImageField = {};
for (const cur of imageField) {
const url = await uploadImage(submitData.general[cur][0]);
acc[cur] = url;
}
console.log(uploadImageField);
}

How do I make a long list of http calls in serial?

I'm trying to only make one http call at time but when I log the response from getUrl they are piling up and I start to get 409s (Too many requests)
function getUrl(url, i, cb) {
const fetchUrl = `https://api.scraperapi.com?api_key=xxx&url=${url.url}`;
fetch(fetchUrl).then(async res => {
console.log(fetchUrl, 'fetched!');
if (!res.ok) {
const err = await res.text();
throw err.message || res.statusText;
}
url.data = await res.text();
cb(url);
});
}
let requests = urls.map((url, i) => {
return new Promise(resolve => {
getUrl(url, i, resolve);
});
});
const all = await requests.reduce((promiseChain, currentTask) => {
return promiseChain.then(chainResults =>
currentTask.then(currentResult => [...chainResults, currentResult]),
);
}, Promise.resolve([]));
Basically I don't want the next http to start until the previous one has finished. Otherwise I hammer their server.
BONUS POINTS: Make this work with 5 at a time in parallel.

Since you're using await, it would be a lot easier to use that everywhere instead of using confusing .thens with reduce. It'd also be good to avoid the explicit Promise construction antipattern. This should do what you want:
const results = [];
for (const url of urls) {
const response = await fetch(url);
if (!response.ok) {
throw new Error(response); // or whatever logic you need with errors
}
results.push(await response.text());
}
Then your results variable will contain an array of response texts (or an error will have been thrown, and the code won't reach the bottom).
The syntax for an async function is an async keyword before the argument list, just like you're doing in your original code:
const fn = async () => {
const results = [];
for (const url of urls) {
const response = await fetch(url);
if (!response.ok) {
throw new Error(response); // or whatever logic you need with errors
}
results.push(await response.text());
}
// do something with results
};
To have a limited number of requests at a time, make a queue system - when a request completes, recursively call a function that makes another request, something like:
const results = [];
const queueNext = async () => {
if (!urls.length) return;
const url = urls.shift();
const response = await fetch(url);
if (!response.ok) {
throw new Error(response); // or whatever logic you need with errors
}
results.push(await response.text());
await queueNext();
}
await Promise.all(Array.from({ length: 5 }, queueNext));
// do something with results

You cannot use Array methods to sequentually run async operations because array methods are all synchronous.
The easiest way to achieve sequential async tasks is through a loop. Otherwise, you will need to write a custom function to imitate a loop and run .then after a async task ends, which is quite troublesome and unnecessary.
Also, fetch is already returning a Promise, so you don't have to create a Promise yourself to contain that promise returned by fetch.
The code below is a working example, with small changes to your original code (see comments).
// Fake urls for example purpose
const urls = [{ url: 'abc' }, { url: 'def', }, { url: 'ghi' }];
// To imitate actual fetching
const fetch = (url) => new Promise(resolve => {
setTimeout(() => {
resolve({
ok: true,
text: () => new Promise(res => setTimeout(() => res(url), 500))
});
}, 1000);
});
function getUrl(url, i, cb) {
const fetchUrl = `https://api.scraperapi.com?api_key=xxx&url=${url.url}`;
return fetch(fetchUrl).then(async res => { // <-- changes here
console.log(fetchUrl, 'fetched!');
if (!res.ok) {
const err = await res.text();
throw err.message || res.statusText;
}
url.data = await res.text();
return url; // <--- changes here
});
}
async function getAllUrls(urls){
const result = [];
for (const url of urls){
const response = await getUrl(url);
result.push(response);
}
return result;
}
getAllUrls(urls)
.then(console.log);

async/await is perfect for this.
Assuming you have an array of URLs as strings:
let urls = ["https://example.org/", "https://google.com/", "https://stackoverflow.com/"];
You simply need to do:
for (let u of urls) {
await fetch(u).then(res => {
// Handle response
}).catch(e => {
// Handle error
});
}
The loop will not iterate until the current fetch() has resolved, which will serialise things.
The reason array.map doesn't work is as follows:
async function doFetch(url) {
return await fetch(url).then(res => {
// Handle response
}).catch(e => {
// Handle error
});
}
let mapped = urls.map(doFetch);
is equivalent to:
let mapped;
for (u of urls) {
mapped.push(doFetch(u));
}
This will populate mapped with a bunch of Promises immediately, which is not what you want. The following is what you want:
let mapped;
for (u of urls) {
mapped.push(await doFetch(u));
}
But this is not what array.map() does. Therefore using an explicit for loop is necessary.

Many people provided answers using for loop. But in some situation await in for loop is not welcome, for example, if you are using Airbnb style guide.
Here is a solution using recursion.
// Fake urls for example purpose
const urls = [{ url: 'abc' }, { url: 'def', }, { url: 'ghi' }];
async function serialFetch(urls) {
return await doSerialRecursion(
async (url) => {
return result = await fetch(url)
.then((response) => {
// handle response
})
.catch((err) => {
// handle error
});
},
urls,
0
);
}
async function doSerialRecursion(fn, array, startIndex) {
if (!array[startIndex]) return [];
const currResult = await fn(array[startIndex]);
return [currResult, ...(await doSerialRecursion(array, fn, startIndex + 1))];
}
const yourResult = await serialFetch(urls);
The doSerialRecursion function will serially execute the function you passed in, which is fetch(url) in this example.

How to resolve a promise in an async function?

I have a lifecycle method componentDidMount that calls upon a recursive async method and I want the recursive function to return a promise after all the data has been fetched.
async componentDidMount() {
let response = await fetch(`${STORY_URL}${this.props.match.params.id}.json`);
let result = await response.json();
totalComments = result.descendants;
await this.fetchComments(result.kids, MARGIN);
this.setState({
by: result.by,
time: calculateTimeDifference(result.time)
});
}
and the function being called is
async fetchComments(comment, margin) {
return new Promise((resolve, reject) => {
comment.map(async commentId => {
let response = await fetch(`${STORY_URL}${commentId}.json`);
let result = await response.json();
comments.push({
by: result.by,
margin: margin
});
if (comments.length === totalComments + 1) resolve();
if (result.kids !== undefined) this.fetchComments(result.kids, margin * 2);
});
});
}
but the resolve method is not returning back to the componentDidMount before the setState. I checked by console logging. I don't know what I am doing wrong

You are overcomplicating things. Use Promise.all:
async fetchComments(comments, margin) {
// Collect all the results here, otgerwise we would have to flatMap the promises which is more complicated
const result = [];
// Make sure that all comments were processed before returning
await Promise.all( comments.map(async commentId => {
const response = await fetch(`${STORY_URL}${commentId}.json`);
const { kids, by } = await response.json();
if(kids) {
// Get all children and append them to the results, right after the children
const children = await fetchComments(kids, margin * 2);
result.push({ by, margin }, ...children);
} else {
// Otherwise just append this node
result.push({ by, margin });
}
}));
return result;
}
If the order matters, you have to flatten the result of Promise.all:
async fetchComments(comments, margin) {
const result = await Promise.all(comments.map( async commentID => {
const response = await fetch(STORY_URL + commentID + ".json");
const { by, kids } = await response.json();
const result = [{ by, margin }];
if(kids) result.push(... await fetchComments(kids, margin * 2));
return result;
}));
// Flatten the results
return [].concat(...result);
}