I have a lifecycle method componentDidMount that calls upon a recursive async method and I want the recursive function to return a promise after all the data has been fetched.
async componentDidMount() {
let response = await fetch(`${STORY_URL}${this.props.match.params.id}.json`);
let result = await response.json();
totalComments = result.descendants;
await this.fetchComments(result.kids, MARGIN);
this.setState({
by: result.by,
time: calculateTimeDifference(result.time)
});
}
and the function being called is
async fetchComments(comment, margin) {
return new Promise((resolve, reject) => {
comment.map(async commentId => {
let response = await fetch(`${STORY_URL}${commentId}.json`);
let result = await response.json();
comments.push({
by: result.by,
margin: margin
});
if (comments.length === totalComments + 1) resolve();
if (result.kids !== undefined) this.fetchComments(result.kids, margin * 2);
});
});
}
but the resolve method is not returning back to the componentDidMount before the setState. I checked by console logging. I don't know what I am doing wrong
You are overcomplicating things. Use Promise.all:
async fetchComments(comments, margin) {
// Collect all the results here, otgerwise we would have to flatMap the promises which is more complicated
const result = [];
// Make sure that all comments were processed before returning
await Promise.all( comments.map(async commentId => {
const response = await fetch(`${STORY_URL}${commentId}.json`);
const { kids, by } = await response.json();
if(kids) {
// Get all children and append them to the results, right after the children
const children = await fetchComments(kids, margin * 2);
result.push({ by, margin }, ...children);
} else {
// Otherwise just append this node
result.push({ by, margin });
}
}));
return result;
}
If the order matters, you have to flatten the result of Promise.all:
async fetchComments(comments, margin) {
const result = await Promise.all(comments.map( async commentID => {
const response = await fetch(STORY_URL + commentID + ".json");
const { by, kids } = await response.json();
const result = [{ by, margin }];
if(kids) result.push(... await fetchComments(kids, margin * 2));
return result;
}));
// Flatten the results
return [].concat(...result);
}
Related
I have node JS api server and I'm having issues with correct chaining of the Promises:
app.post(
"/api/tasks",
async function (_req, res) {
const newArray = [{ MyTasks: [] }];
const getOne = async (owner, taskID) => {
return await getOneDocument(owner, taskID).then((result) => {
console.log("get one doc", result);
return result;
});
};
// first promise
let toApproveTasks = await getToApproveTasks(_req.body.userID);
console.log("1", toApproveTasks);
// loop trough the result of 1st promise and run async function for each
const arrayToDoc = async (array) => {
array.TasksToApprove.forEach(async (element) => {
let objToPush = await getOne(element.Owner, element.TaskID);
console.log("1.5", objToPush);
newArray.MyTasks.push(objToPush);
});
};
// second promise
await arrayToDoc(toApproveTasks);
console.log("2", newArray);
// third promise
let finalResult = await parseCosmosOutput(newArray);
console.log("3", finalResult);
res.status(200).send(finalResult);
}
);
What I get in console is :
1 [Object] - all good
Emppty Array
Empty Array
get one doc {object} - all good
1.5 {object} - all good
How would I make sure when I loop over result of 1st promise my code awaits async function and pushes to newArray results ?
Use For..of instead of forEach inside arrayToDoc function
E.g
const arrayToDoc = async (array) => {
for(let element of array.TasksToApprove){
let objToPush = await getOne(element.Owner, element.TaskID);
console.log("1.5", objToPush);
newArray.MyTasks.push(objToPush);
}
};
The value of videoProgress is not returned in the final object document, i want videoProgress to be returned along with the documentList item, but i am not getting it, i have set item.videoProgress too but not getting it in response:
const document = await Promise.all(documentList.map(async (item) => {
const videoCount = await studentProgressModel.find({ gradeId: item.gradeId, userId : item._id, type: 'VIDEO'}).countDocuments();
const totalVideoCount = await videoModel.find({ gradeId: item.gradeId, mediumId: item.mediumId, status: 'ACTIVE' }).countDocuments();
let videoPercentage = 0;
if(totalVideoCount > 0) {
videoPercentage = (100 * videoCount) / totalVideoCount;
}
item.videoProgress = videoPercentage;
console.log('item', item);
return item;
}));
The issue here is not what anyone in the comments above is discussing, the issue is that you are not actually returning a promise from your documentList.map, but instead are just writing an async function there. Array.map will not wait for any asynchronous behavior, so as soon as it hits an await it returns an empty result.
This can be fixed by instead returning promises in your map:
const document = await Promise.all(documentList.map((item) => { // Removed async here, map is synchronous
// Here we *immediately* return a promise so that map can return that to the promise.all
return new Promise(async (resolve) => {
// Now we can do all our async stuff and promise.all will wait for it :)
const videoCount = await studentProgressModel.find({ gradeId: item.gradeId, userId : item._id, type: 'VIDEO'}).countDocuments();
const totalVideoCount = await videoModel.find({ gradeId: item.gradeId, mediumId: item.mediumId, status: 'ACTIVE' }).countDocuments();
let videoPercentage = 0;
if(totalVideoCount > 0) {
videoPercentage = (100 * videoCount) / totalVideoCount;
}
item.videoProgress = videoPercentage;
console.log('item', item);
return resolve(item); // have to use resolve here with our result to resolve the promise.
});
}));
There are other ways to do this that don't involve an explicit new-promise wrapper, but I figured it made the promise behavior more obvious
Edit: Here's an example which can be run directly here in your browser, I have only replaced the queries with async random number generators.
function randomInteger() {
return new Promise((resolve) => {
setTimeout(() => {
return resolve(Math.floor(Math.random() * 100));
}, 1);
});
}
async function main() {
let documentList = [{}, {}, {}];
const document = await Promise.all(documentList.map((item) => {
return new Promise(async (resolve) => {
const videoCount = await randomInteger();
const totalVideoCount = await randomInteger();
let videoPercentage = 0;
if(totalVideoCount > 0) {
videoPercentage = (100 * videoCount) / totalVideoCount;
}
item.videoProgress = videoPercentage;
console.log('item', item);
return resolve(item);
});
}));
console.log('Document results:', document);
}
main();
I'm trying to only make one http call at time but when I log the response from getUrl they are piling up and I start to get 409s (Too many requests)
function getUrl(url, i, cb) {
const fetchUrl = `https://api.scraperapi.com?api_key=xxx&url=${url.url}`;
fetch(fetchUrl).then(async res => {
console.log(fetchUrl, 'fetched!');
if (!res.ok) {
const err = await res.text();
throw err.message || res.statusText;
}
url.data = await res.text();
cb(url);
});
}
let requests = urls.map((url, i) => {
return new Promise(resolve => {
getUrl(url, i, resolve);
});
});
const all = await requests.reduce((promiseChain, currentTask) => {
return promiseChain.then(chainResults =>
currentTask.then(currentResult => [...chainResults, currentResult]),
);
}, Promise.resolve([]));
Basically I don't want the next http to start until the previous one has finished. Otherwise I hammer their server.
BONUS POINTS: Make this work with 5 at a time in parallel.
Since you're using await, it would be a lot easier to use that everywhere instead of using confusing .thens with reduce. It'd also be good to avoid the explicit Promise construction antipattern. This should do what you want:
const results = [];
for (const url of urls) {
const response = await fetch(url);
if (!response.ok) {
throw new Error(response); // or whatever logic you need with errors
}
results.push(await response.text());
}
Then your results variable will contain an array of response texts (or an error will have been thrown, and the code won't reach the bottom).
The syntax for an async function is an async keyword before the argument list, just like you're doing in your original code:
const fn = async () => {
const results = [];
for (const url of urls) {
const response = await fetch(url);
if (!response.ok) {
throw new Error(response); // or whatever logic you need with errors
}
results.push(await response.text());
}
// do something with results
};
To have a limited number of requests at a time, make a queue system - when a request completes, recursively call a function that makes another request, something like:
const results = [];
const queueNext = async () => {
if (!urls.length) return;
const url = urls.shift();
const response = await fetch(url);
if (!response.ok) {
throw new Error(response); // or whatever logic you need with errors
}
results.push(await response.text());
await queueNext();
}
await Promise.all(Array.from({ length: 5 }, queueNext));
// do something with results
You cannot use Array methods to sequentually run async operations because array methods are all synchronous.
The easiest way to achieve sequential async tasks is through a loop. Otherwise, you will need to write a custom function to imitate a loop and run .then after a async task ends, which is quite troublesome and unnecessary.
Also, fetch is already returning a Promise, so you don't have to create a Promise yourself to contain that promise returned by fetch.
The code below is a working example, with small changes to your original code (see comments).
// Fake urls for example purpose
const urls = [{ url: 'abc' }, { url: 'def', }, { url: 'ghi' }];
// To imitate actual fetching
const fetch = (url) => new Promise(resolve => {
setTimeout(() => {
resolve({
ok: true,
text: () => new Promise(res => setTimeout(() => res(url), 500))
});
}, 1000);
});
function getUrl(url, i, cb) {
const fetchUrl = `https://api.scraperapi.com?api_key=xxx&url=${url.url}`;
return fetch(fetchUrl).then(async res => { // <-- changes here
console.log(fetchUrl, 'fetched!');
if (!res.ok) {
const err = await res.text();
throw err.message || res.statusText;
}
url.data = await res.text();
return url; // <--- changes here
});
}
async function getAllUrls(urls){
const result = [];
for (const url of urls){
const response = await getUrl(url);
result.push(response);
}
return result;
}
getAllUrls(urls)
.then(console.log);
async/await is perfect for this.
Assuming you have an array of URLs as strings:
let urls = ["https://example.org/", "https://google.com/", "https://stackoverflow.com/"];
You simply need to do:
for (let u of urls) {
await fetch(u).then(res => {
// Handle response
}).catch(e => {
// Handle error
});
}
The loop will not iterate until the current fetch() has resolved, which will serialise things.
The reason array.map doesn't work is as follows:
async function doFetch(url) {
return await fetch(url).then(res => {
// Handle response
}).catch(e => {
// Handle error
});
}
let mapped = urls.map(doFetch);
is equivalent to:
let mapped;
for (u of urls) {
mapped.push(doFetch(u));
}
This will populate mapped with a bunch of Promises immediately, which is not what you want. The following is what you want:
let mapped;
for (u of urls) {
mapped.push(await doFetch(u));
}
But this is not what array.map() does. Therefore using an explicit for loop is necessary.
Many people provided answers using for loop. But in some situation await in for loop is not welcome, for example, if you are using Airbnb style guide.
Here is a solution using recursion.
// Fake urls for example purpose
const urls = [{ url: 'abc' }, { url: 'def', }, { url: 'ghi' }];
async function serialFetch(urls) {
return await doSerialRecursion(
async (url) => {
return result = await fetch(url)
.then((response) => {
// handle response
})
.catch((err) => {
// handle error
});
},
urls,
0
);
}
async function doSerialRecursion(fn, array, startIndex) {
if (!array[startIndex]) return [];
const currResult = await fn(array[startIndex]);
return [currResult, ...(await doSerialRecursion(array, fn, startIndex + 1))];
}
const yourResult = await serialFetch(urls);
The doSerialRecursion function will serially execute the function you passed in, which is fetch(url) in this example.
I'm using async await inside of for loop as below.
for (let i = 0; i < result.length; i += 1) {
try{
const client = await axios.get(
`${process.env.user}/client/${result[i].id}`
);
} catch(error){
console.log(error)
}
if (client.data.success === true) {
result[i].Name = rider.data.client.Name;
result[i].PhoneNumber = rider.data.client.Number;
}
}
But I want to make this using 'new Promise' and 'promiss.all' to make it asynclously.
But I don'k know how to make this correctly doing error handle well.
Could you recommend some advice for this? Thank you for reading it.
This can be a basic solution, i think
let playList = []
for (let i = 0; i < result.length; i += 1) {
playList.push(axios.get(
`${process.env.user}/client/${result[i].id}`
).then(res => {
if (res.data.success === true) {
result[i].Name = rider.data.client.Name;
result[i].PhoneNumber = rider.data.client.Number;
}
}).catch(ex => console.log(ex)));
}
await Promise.all(playList)
This can also be done by using a foreach loop.
The for/foreach loop can be simplified by using a map function.
Js map function is equivalent of c# select linq function.
The fat arrow in js map function is not bound to return a value unlike c# select inner function which must return a value.
await Promise.all(result.map(async r => {
let client;
try {
client = await axios.get(`${process.env.user}/client/${r.id}`);
} catch (error) {
console.log(error)
}
if (client.data.success === true) {
r.Name = rider.data.client.Name;
r.PhoneNumber = rider.data.client.Number;
}
}));
Try this
var promises = result.map(r => axios.get(`${process.env.user}/client/${r.id}`);
Promise.all(promises).then(function(values) {
console.log('All promises done');
});
The idea is that if you are awaiting something, that is promise, you can await it, or call it to get promise
Example:
function Foo()
{
return new Promise(...); // Promise of int for example
}
you can do
var p = Foo(); //you will get promise
Or
var v = await Foo(); // you will get int value when promise resolved
This is how you do it with async/await + Promise.all:
const myResult = await Promise.all(result.map(({ id }) => {
return axios.get(`${process.env.user}/client/${id}`);
}));
// deal with the result of those requests
const parsed = myResult.map(data => /* your code here */);
Here is an example using Array.map to call your function along with Promise.all. I wrapped the axios request in a function so if one of your request fails, it wont stop every other requests. If you don't mind stopping when you got an issue, look at others answers to your question.
function fakeRequest() {
return new Promise((resolve, reject) => {
setTimeout(() => {
resolve({
data: {
success: true,
client: {
Name: 'Todd',
Number: 5,
},
},
});
}, 300);
});
}
(async() => {
const result = [{}, {}, {}];
await Promise.all(result.map(async(x, xi) => {
try {
const client = await fakeRequest();
if (client.data.success === true) {
result[xi].Name = client.data.client.Name;
result[xi].PhoneNumber = client.data.client.Number;
}
} catch (err) {
console.log(err)
}
}));
console.log(result);
})();
I'm trying to skip one object from an array objects based on a async operator. I've tried following cases, but getting a Type error.
Tried Method 1
newObjectArray = await Promise.all(objectAray.reduce(async (result, el) => {
const asyncResult = await someAsyncTask(el);
if (asyncResult) {
result.push(newSavedFile);
}
return result;
}, []));
Tried Method 2
newObjectArray = await Promise.all(objectAray.reduce(async (prevPromise, el) => {
const collection = await prevPromise;
const asyncResult = await someAsyncTask(el);
if (asyncResult) {
prevPromise.push(newSavedFile);
}
collection.push(newSavedFile);
return collection;
}, Promise.resolve([])));
Error
'TypeError: #<Promise> is not iterable',
' at Function.all (<anonymous>)',
In your first try, result is a promise as all async functions evaluate to a promise when called, so you have to await result before you can push to the array, and then you don't need the Promise.all:
newObjectArray = await objectAray.reduce(async (result, el) => {
const asyncResult = await someAsyncTask(el);
if (asyncResult) {
(await result).push(newSavedFile);
}
return result;
}, []);
But I'd guess that it is way faster to just filter afterwards:
newObjectArray = (await Promise.all(objArray.map(someAsyncTask))).filter(el => el);