I'm fetching product's data from a bunch of URLs. The result of these promises (along with product details) may contain a key called nextPage, which has a URL as value. I need to fetch data from that URL too. How can I wait for all the original set of promises + dynamic promises attached based on nextPage key ?
Currently I'm trying the following:
const allUrls = ['url1', 'url2', 'url3'];
const promiseArray = [];
const getData = (url) => {
const p = axios.get(url).then((data) => {
if (data.nextPage) {
// Push this into promises so that the data is not lost
promiseArray.push(Promise.resolve(data));
// There are more pages to fetch data from
return getData(data.nextPage);
}
// No further pages, so return the result
return data;
});
return p;
}
allUrls.forEach((url) => {
const p = getData(url);
promiseArray.push(p);
});
Promise.all(promiseArray).then((data) => {
// Error: data only has the value of allUrls, not dynamically added data
})
As I have put a comment, when the Promise.all resolves, I only get the data of original URLs put in allUrls, not the dynamically added promises.
The problem is that by the time Promise.all is executed, promiseArray will only contain the promises for the product's data, not the details. You'd need to await each product data promise, and then push the resulting promise to the array (note that I'm using async/await as it allows for cleaner code):
for (const url of allUrls) {
const p = await getData(url);
promiseArray.push(p);
}
const result = await Promise.all(promiseArray);
console.log(result);
Related
This question already has answers here:
Why is my variable unaltered after I modify it inside of a function? - Asynchronous code reference
(7 answers)
Closed 11 months ago.
I have an api route that needs to take data from two sources, merge the data together into one object and then return. The issue I am having is I'm basically stuck in async/await hell and when pushing to a second array within the .then() block, the second array named
clone returns []. How can I make an api request, merge the data and return to the requester as needed?
Fetch code:
export default async function getProduct(product_id) {
const product = await fetch(
`${process.env.PRIVATE_APP_URL}/products/${product_id}.json`,
{
method: "GET",
headers: {
"Content-Type": "application/json",
},
}
).then((result) => {
return result.json();
});
return product.product;
}
API Handler:
const recharge_subscription_res = await rechargeAPI(
"GET",
`https://api.rechargeapps.com/subscriptions?customer_id=${recharge_customer.id}`
);
const closest_array = recharge_subscription_res.subscriptions.filter(
(e) => e.next_charge_scheduled_at == closest_date
);
let clone = [];
closest_array.forEach((element) => {
getProduct(element.shopify_product_id).then((product) => {
element.shopify_product_handle = product.handle;
element.shopify_product_image_url = product.image.src;
clone.push(element);
});
});
console.log(clone);
clone should log as an array of objects like closest_array, but instead logs as just an empty array. This isn't exactly like the other seemingly duplicate questions because typically their feature doesn't require sending the promise's data back to an external source. Most questions are related to the front end of things. My situation is with an Express.js API. Any help would be appreciated.
The original promise spec used .then(), and the new syntax hides then's with await. Style-wise, it makes sense to choose just one style and go with it.
In either style, there's a little challenge having to do with creating many promises in a loop. The js iteration functions (like map and forEach) take synchronous functions. The most common design is to create a collection of promises in a synchronous loop, then run them concurrently with Promise.all(). Taking both ideas into account...
You could (but don't have to) rewrite your network request like this...
// since we decorated "async" let's use await...
export default async function getProduct(product_id) {
const url = `${process.env.PRIVATE_APP_URL}/products/${product_id}.json`;
const options = { method: "GET", headers: { "Content-Type": "application/json" }};
const result = await fetch(url, options);
const product = await result.json();
return product.product;
}
await is not permitted at the top-level; it may be used only within an async function. Here, I'll make up a name and guess about the parameter
async function rechargeAndLookupProduct(recharge_customer) {
const base = 'https://api.rechargeapps.com/subscriptions';
const query = `customer_id=${recharge_customer.id}`;
const recharge_subscription_res = await rechargeAPI("GET",`${base}?${query}`);
const closest_array = recharge_subscription_res.subscriptions.filter(e =>
e.next_charge_scheduled_at == closest_date
);
// here's the important part: collect promises synchronously
// execute them together with Promise.all()
const promises = closest_array.map(element => {
return getProduct(element.shopify_product_id)
});
const allProducts = await Promise.all(promises);
// allProducts will be an array of objects that the promises resolved to
const clones = allProducts.map((product, i) => {
// use Object.assign so we'll really have a "clone"
let closest = Object.assign({}, closest_array[i]);
closest.shopify_product_handle = product.handle;
closest.shopify_product_image_url = product.image.src;
return closest;
});
// if I didn't make any typos (which I probably did), then
// clones ought to contain the result you expect
console.log(clones);
}
Your code has a flaw (in the section shown below). You have a dangling promise that you forgot to await or return.
When you log clone, none of the async getProduct operations have completed yet, and none of the elements have been pushed.
let clone = [];
closest_array.forEach((element) => {
getProduct(element.shopify_product_id).then((product) => {
element.shopify_product_handle = product.handle;
element.shopify_product_image_url = product.image.src;
clone.push(element);
}); // FLAW: dangling .then
});
console.log(clone); // FLAW: clone is not ready yet.
I would set it up more like this:
let clone = await Promise.all(closest_array.map((element) =>
getProduct(element.shopify_product_id).then((product) => {
element.shopify_product_handle = product.handle;
element.shopify_product_image_url = product.image.src;
return element;
})
));
console.log(clone);
It's a little sketchy to modify element the way you are (I wouldn't), but this way the getProduct calls are all in flight together for maximum efficiency. Promise.all handles awaiting all the promises and putting each's result into an array of results, which you can then await as a single promise since the calling function is async.
I'm trying to upload several pictures to Firebase Storage and get their URLs to add these as a field in a gym's profile. However, with the code I came up with, always one URL is not returned. If I upload 3 pictures, 3 pictures appear in Storage but I only get 2 URLs. How do I fix that?
const promises = [];
const URLarray = [];
imageAsFile.forEach(img => {
const uploadTask = storage.ref().child(`/photos/${doc.id}/${img.name}`).put(img);
promises.push(uploadTask);
uploadTask.then((uploadTaskSnapshot) => {
return uploadTaskSnapshot.ref.getDownloadURL();
}).then((pic) => {
const addPics = db.collection("gyms").doc(doc.id)
promises.push(addPics);
addPics.update({gymPhoto: firebase.firestore.FieldValue.arrayUnion(pic)});
// URLarray.push(pic)
})
});
Promise.all(promises).then(() => {
// console.log(URLarray)
alert("Done");
})
The following should work (untested):
const promises = [];
imageAsFile.forEach(img => {
const uploadTask = storage.ref().child(`/photos/${doc.id}/${img.name}`).put(img);
promises.push(uploadTask);
});
Promise.all(promises)
.then(uploadTaskSnapshotsArray => {
const promises = [];
uploadTaskSnapshotsArray.forEach(uploadTaskSnapshot => {
promises.push(uploadTaskSnapshot.ref.getDownloadURL());
});
return Promise.all(promises);
})
.then(urlsArray => {
const docRef = db.collection("gyms").doc(doc.id);
return docRef.update({ gymPhoto: firebase.firestore.FieldValue.arrayUnion(...urlsArray) });
})
Promise.all() returns a single promise: This promise is fulfilled with an Array containing all the resolved values in the Array passed as the argument.
So, if you call Promise.all() with the Array of uploadTasks, when the returned promise is fulfilled you will get an Array of uploadTaskSnapshots.
Then you need to use this Array to get an Array of URLs, again by using Promise.all().
Then, when you get the Array of URLs, you need to update only once the document with id = doc.id. So here you don't need Promise.all(). What you need to do is to pass several values to the FieldValue.arrayUnion() method. For that you need to use the Spread operator, as follows:
docRef.update({ gymPhoto: firebase.firestore.FieldValue.arrayUnion(...urlsArray) });
I'm fetching my user data and the map function is called several times for each user. I want to wait until all data was pushed to the array and then manipulate the data. I tried using Promise.all() but it didn't work.
How can I wait for this map function to finish before continuing?
Needless to say that the array user_list_temp is empty when printed inside the Promise.all().
const phone_list_promise_1 = await arrWithKeys.map(async (users,i) => {
return firebase.database().ref(`/users/${users}`)
.on('value', snapshot => {
user_list_temp.push(snapshot.val());
console.log(snapshot.val());
})
}
);
Promise.all(phone_list_promise_1).then( () => console.log(user_list_temp) )
I changed the code to this but I still get a wrong output
Promise.all(arrWithKeys.map(async (users,i) => {
const eventRef = db.ref(`users/${users}`);
await eventRef.on('value', snapshot => {
const value = snapshot.val();
console.log(value);
phone_user_list[0][users].name = value.name;
phone_user_list[0][users].photo = value.photo;
})
console.log(phone_user_list[0]);
user_list_temp.push(phone_user_list[0]);
}
));
console.log(user_list_temp); //empty array
}
It is possible to use async/await with firebase
This is how I usually make a Promise.all
const db = firebase.database();
let user_list_temp = await Promise.all(arrWithKeys.map(async (users,i) => {
const eventRef = db.ref(`users/${users}`);
const snapshot = await eventref.once('value');
const value = snapshot.value();
return value;
})
);
This article gives a fairly good explanation of using Promise.all with async/await https://www.taniarascia.com/promise-all-with-async-await/
Here is how I would refactor your new code snippet so that you are not mixing promises and async/await
let user_list_temp = await Promise.all(arrWithKeys.map(async (users,i) => {
const eventRef = db.ref(`users/${users}`);
const snapshot= await eventRef.once('value');
const value = snapshot.val();
console.log(value);
phone_user_list[0][users].name = value.name; // should this be hardcoded as 0?
phone_user_list[0][users].photo = value.photo; // should this be hardcoded as 0?
console.log(phone_user_list[0]);
return phone_user_list[0]; // should this be hardcoded as 0?
})
);
console.log(user_list_temp);
Here is a simple example that uses fetch, instead of firebase:
async componentDidMount () {
let urls = [
'https://jsonplaceholder.typicode.com/todos/1',
'https://jsonplaceholder.typicode.com/todos/2',
'https://jsonplaceholder.typicode.com/todos/3',
'https://jsonplaceholder.typicode.com/todos/4'
];
let results = await Promise.all(urls.map(async url => {
const response = await fetch(url);
const json = await response.json();
return json;
}));
alert(JSON.stringify(results))
}
If I understand your question correctly, you might consider revising your code to use a regular for..of loop, with a nested promise per user that resolves when the snapshot/value for that user is available as shown:
const user_list_temp = [];
/*
Use regular for..of loop to iterate values
*/
for(const user of arrWithKeys) {
/*
Use await on a new promise object for this user that
resolves with snapshot value when value recieved for
user
*/
const user_list_item = await (new Promise((resolve) => {
firebase.database()
.ref(`/users/${users}`)
.on('value', snapshot => {
/*
When value recieved, resolve the promise for
this user with val()
*/
resolve(snapshot.val());
});
}));
/*
Add value for this user to the resulting user_list_item
*/
user_list_temp.push(user_list_item);
}
console.log(user_list_temp);
This code assumes that the enclosing function is defined as an asynchronous method with the async keyword, seeing that the await keyword is used in the for..of loop. Hope that helps!
What i want to do is to upload file on server, then get URL of uploaded file and preview it. Files can be more than one. For that purpose i have written following code:
let filesURL=[];
let promises=[];
if(this.state.files_to_upload.length>0) {
for(let i=0; i<this.state.files_to_upload.length; i++) {
promises.push(this.uploadFilesOnServer(this.state.files_to_upload[i]))
}
Promise.all(promises).then(function(result){
console.log(result);
result.map((file)=>{
filesURL.push(file);
});
});
console.log(filesURL);
}
const uploadedFilesURL=filesURL;
console.log(uploadedFilesURL);
console.log(filesURL); give me the values returned by Promise.all.
And i want to use these values only when Promise.all completes properly. But, i am facing problem that lines console.log(uploadedFilesURL); excutes first irrespective of Promise.all and give me undefined values.I think i am not using promises correctly, can anyone please help me?
uploadFileOnServer code is:
uploadFilesOnServer(file)
{
let files=[];
let file_id='';
const image=file;
getImageUrl().then((response) => {
const data = new FormData();
data.append('file-0', image);
const {upload_url} = JSON.parse(response);
console.log(upload_url);
updateProfileImage(upload_url, data).then ((response2) => {
const data2 = JSON.parse(response2);
file_id=data2;
console.log(file_id);
files.push(file_id);
console.log(files);
});
});
return files;
}
No, promise is asynchronous and as such, doesn't work the way you think. If you want to execute something after a promise completed, you must put it into the promise's then callback. Here is the example based on your code:
uploadFilesOnServer(file) {
let files=[];
let file_id='';
const promise = getImageUrl()
.then((imageUrlResponse) => {
const data = new FormData();
data.append('file-0', file);
const { upload_url } = JSON.parse(imageUrlResponse);
console.log(upload_url);
return updateProfileImage(upload_url, data);
})
.then ((updateImageResponse) => {
file_id= JSON.parse(updateImageResponse);
console.log(file_id);
files.push(file_id);
console.log(files);
return files;
});
return promise;
}
let filesPromise = Promise.resolve([]);
if(this.state.files_to_upload.length > 0) {
const promises = this.state.files_to_upload.map((file) => {
return this.uploadFilesOnServer(file);
});
filesPromise = Promise.all(promises).then((results) => {
console.log(results);
return [].concat(...results);
});
}
// This is the final console.log of you (console.log(uploadedFilesURL);)
filesPromise.then((filesUrl) => console.log(filesUrl));
A good book to read about ES6 in general and Promises in particular is this book Understanding ECMAScript 6 - Nicholas C. Zakas
Edit:
Here is an simple explanation of the example code:
The uploadFilesOnServer is a function that takes a file, upload it and will return the file URL when the upload completes in the future in the form of a promise. The promise will call its then callback when it gets the url.
By using the map function, we create a list of url promises, the results we've got from executing uploadFilesOnServer on each file in the list.
The Promise.all method waits for all the promises in the list to be completed, joins the list of url results and create a promise with the result which is the list of urls. We need this because there is no guarantee that all of the promises will complete at once, and we need to gather all the results in one callback for convenience.
We get the urls from the then callback.
You have to do this on the .then part of your Promise.all()
Promise.all(promises)
.then(function(result){
console.log(result);
result.map((file)=>{
filesURL.push(file);
});
return true; // return from here to go to the next promise down
})
.then(() => {
console.log(filesURL);
const uploadedFilesURL=filesURL;
console.log(uploadedFilesURL);
})
This is the way async code works. You cannot expect your console.log(filesURL); to work correctly if it is being called syncronously after async call to fetch files from server.
Regarding to your code there are several problems:
1.uploadFilesOnServer must return Promise as it is async. Therefore:
uploadFilesOnServer(file)
{
let files=[];
let file_id='';
const image=file;
return getImageUrl().then((response) => {
const data = new FormData();
data.append('file-0', image);
const {upload_url} = JSON.parse(response);
console.log(upload_url);
updateProfileImage(upload_url, data).then ((response2) => {
const data2 = JSON.parse(response2);
file_id=data2;
console.log(file_id);
files.push(file_id);
console.log(files);
return files;
});
});
}
2.Inside your main function body you can assess results of the Promise.all execution only in its respective then handler.
As a side note I would recomment you to use es7 async/await features with some transpilers like babel/typescript. This will greatly reduce the nesting/complications of writing such async code.
I am creating an API that when GET, a series of calls to the News API are made, news article titles are extracted into a giant string, and that string is processed into an object to be delivered to a wordcloud on the front-end. So far, I've been able to use underscore's _.after and request-promise to make my app wait till all API calls have completed before calling processWordBank() which takes the giant string and cleans it up into an object. However, once processWordBank() is called, I don't understand where the flow of the program is. Ideally, processWordBank() returns obj to cloudObj in the router, so that the obj can be passed to res.json() and spit out as the response. I believe my use of _.after has put me in a weird situation, but it's the only way I've been able to get async calls to finish before proceeding to next desired action. Any suggestions?
(I've tried to leave out all unnecessary code but let me know if this is insufficient)
// includes...
var sourceString = ""
// router
export default ({ config }) => {
let news = Router()
news.get('/', function(req, res){
var cloudObj = getSources()
res.json({ cloudObj })
})
return news
}
// create list of words (sourceString) by pulling news data from various sources
function getSources() {
return getNewsApi()
}
// NEWS API
// GET top 10 news article titles from News API (news sources are determined by the values of newsApiSource array)
function getNewsApi() {
var finished = _.after(newsApiSource.length, processWordBank)
for(var i = 0; i < newsApiSource.length; i++) {
let options = {
uri: 'https://newsapi.org/v1/articles?source=' + newsApiSource[i] + '&sortBy=' + rank + '&apiKey=' + apiKey,
json: true
}
rp(options)
.then(function (res) {
let articles = res.articles // grab article objects from the response
let articleTitles = " " + _.pluck(articles, 'title') // extract title of each news article
sourceString += " " + articleTitles // add all titles to the word bank
finished() // this async task has finished
})
.catch(function (err) {
console.log(err)
})
}
}
// analyse word bank for patterns/trends
function processWordBank(){
var sourceArray = refineSource(sourceString)
sourceArray = combineCommon(sourceArray)
sourceArray = getWordFreq(sourceArray)
var obj = sortToObject(sourceArray[0], sourceArray[1])
console.log(obj)
return obj
}
A big issue in your asynchronous flow is that you use a shared variable sourceString to handle the results. When you have multiple calls to getNewsApi() your result is not predictable and will not always be the same, because there is no predefined order in which the asynchronous calls are executed. Not only that, but you never reset it, so all subsequent calls will also include the results of the previous calls. Avoid modifying shared variables in asynchronous calls and instead use the results directly.
I've been able to use underscore's _.after and request-promise to make my app wait till all API calls have completed before calling processWordBank()
Although it would possible to use _.after, this can be done very nicely with promises, and since you're already using promises for your requests, it's just a matter of collecting the results from them. So because you want to wait until all API calls are completed you can use Promise.all which returns a promise that resolves with an array of the values of all the promises, once all of them are fulfilled. Let's have a look at a very simple example to see how Promise.all works:
// Promise.resolve() creates a promise that is fulfilled with the given value
const p1 = Promise.resolve('a promise')
// A promise that completes after 1 second
const p2 = new Promise(resolve => setTimeout(() => resolve('after 1 second'), 1000))
const p3 = Promise.resolve('hello').then(s => s + ' world')
const promises = [p1, p2, p3]
console.log('Waiting for all promises')
Promise.all(promises).then(results => console.log('All promises finished', results))
console.log('Promise.all does not block execution')
Now we can modify getNewsApi() to use Promise.all. The array of promises that is given to Promise.all are all the API request you're doing in your loop. This will be created with Array.protoype.map. And also instead of creating a string out of the array returned from _.pluck, we can just use the array directly, so you don't need to parse the string back to an array at the end.
function getNewsApi() {
// Each element is a request promise
const apiCalls = newsApiSource.map(function (source) {
let options = {
uri: 'https://newsapi.org/v1/articles?source=' + source + '&sortBy=' + rank + '&apiKey=' + apiKey,
json: true
}
return rp(options)
.then(function (res) {
let articles = res.articles
let articleTitles = _.pluck(articles, 'title')
// The promise is fulfilled with the articleTitles
return articleTitles
})
.catch(function (err) {
console.log(err)
})
})
// Return the promise that is fulfilled with all request values
return Promise.all(apiCalls)
}
Then we need to use the values in the router. We know that the promise returned from getNewsApi() fulfils with an array of all the requests, which by themselves return an array of articles. That is a 2d array, but presumably you would want a 1d array with all the articles for your processWordBank() function, so we can flatten it first.
export default ({ config }) => {
let news = Router()
new.get('/', (req, res) => {
const cloudObj = getSources()
cloudObj.then(function (apiResponses) {
// Flatten the array
// From: [['source1article1', 'source1article2'], ['source2article1'], ...]
// To: ['source1article1', 'source1article2', 'source2article1', ...]
const articles = [].concat.apply([], apiResponses)
// Pass the articles as parameter
const processedArticles = processWordBank(articles)
// Respond with the processed object
res.json({ processedArticles })
})
})
}
And finally processWordBank() needs to be changed to use an input parameter instead of using the shared variable. refineSource is no longer needed, because you're already passing an array (unless you do some other modifications in it).
function processWordBank(articles) {
let sourceArray = combineCommon(articles)
sourceArray = getWordFreq(sourceArray)
var obj = sortToObject(sourceArray[0], sourceArray[1])
console.log(obj)
return obj
}
As a bonus the router and getNewsApi() can be cleaned up with some ES6 features (without the comments from the snippets above):
export default ({ config }) => {
const news = Router()
new.get('/', (req, res) => {
getSources().then(apiResponses => {
const articles = [].concat(...apiResponses)
const processedArticles = processWordBank(articles)
res.json({ processedArticles })
})
})
}
function getNewsApi() {
const apiCalls = newsApiSource.map(source => {
const options = {
uri: `https://newsapi.org/v1/articles?source=${source}&sortBy=${rank}&apiKey=${apiKey}`,
json: true
}
return rp(options)
.then(res => _.pluck(res.articles, 'title'))
.catch(err => console.log(err))
})
return Promise.all(apiCalls)
}