We Keep Coding

Javascript for loop does not wait for fetch request to complete and moves onto next iteration

I have below code in javascript in which some asynchronous task is being performed:
async function fetchData(id){
for(let i=1;;++i){
res = await fetch(`https://some-api/v1/products/${id}/data?page=${i}`,{//headers here});
res = await res.json();
if(res.length==0) break;
else{ //do some work here and continue for next iteration}
}
}
async function callApi(){
var arr = [//list of id's here to pass to api one by one, almost 100 id's here];
await Promise.all(arr.map(async(e)=>{
await fetchData(e);
}));
}
callApi();
The above code looks fine to me, except that it doesn't work as expected. Ideally, what should happen is that unless one id's call is not completed( unless break condition not satisfies for one id), the for loop should not proceed to next iteration. Rather, I am getting totally different results. The api calls are happening in random order because the loop is not waiting the iteration to complete. My hard requirement is that unless one iteration is not complete, it should not move to next one.
await seems to have no effect here. Please guide me how can I achieve this. I am running out of ideas.
Thank You!

Your arr.map(...) is not awaiting the different fetchData calls before the next map call, so I'd turn this into a specific for loop to be sure it waits:
async function callApi(){
const arr = [...];
for(let i = 0; i < arr.length; i++){
await fetchData(arr[i]);
}
}
or alternatively use a for of
async function callApi(){
const arr = [...];
for(let a of arr){
await fetchData(a);
}
}
The fetchData function also looks like it could use some improvements with error handling, but since you shortened your code quite a bit, I'm assuming there is something like that going on there, too, and your issue is actually with the callApi() code instead, as the fetch and await looks good to me there.

You should decide either to use promises or async await. Don't mix them.
With promises you can always use funky abstractions but with a simple recursive approach you can do like
function fetchData(hds, id, page = 1, pages = []){
return fetch(`https://some-api/v1/products/${id}/data?page=${page}`,hds)
.then(r => r.ok ? r.json() : Promise.reject({status:r.status,pages})
.then(j => fetchData(hds, id, ++page, pages.push(doSomethingWith(j))))
.catch(e => (console.log(e.status), e.pages));
}
So we use recursion to fetch indefinitelly until the API says enough and r.ok is false.
At the callApi side you can use reduce since we have an ids array.
const ids = [/* ids array */],
hds = { /* headers object */ };
function callApi(ids){
return ids.reduce( (p,id) => p.then(_ => fetchData(hds,id))
.then(pages => process(pages))
, Promise.resolve(null)
)
.catch(e => console.log(e));
}
So now both accesses to the id and page data are working asynchronously but only fired once the previous one finishes. Such as
(id=1,page=1) then (id=1,page=2) then (id=1,page=3) then (process 3 pages of id=1) then
(id=2,page=1) then (id=2,page=2) then (process 2 pages of id=2) etc...
While I love the promises, you can also implement the same functionality with the asyc await abstraction. I believe the idea behind the invention of the async await is to mimic sync imperative code. But keep in mind that it's an abstraction over an abstraction and I urge you to learn promises by heart before even attemting to use async await. The general rule is to never mix both in the same code.
Accordingly the above code could have been written as follows by using async await.
async function fetchData(hds, id){
let page = 1,
pages = [],
res;
while(true){
res = await fetch(`https://some-api/v1/products/${id}/data?page=${page++}`,hds);
if (res.ok) pages.push(await res.json())
else return pages;
}
}
Then the callApi function can be implemented in a similar fashion
const ids = [/* ids array */],
hds = { /* headers object */ };
async function callApi(ids){
let pages;
for(let i = 0; i < ids.length; i++){
try {
pages = await fetchData(hds,ids[i]);
await process(pages); // no need for await if the process function is sync
}
catch(e){
console.log(e);
}
}
}

Add key to array of objects in async function [duplicate]

This question already has answers here:
Why is my variable unaltered after I modify it inside of a function? - Asynchronous code reference
(7 answers)
Closed 11 months ago.
I have an api route that needs to take data from two sources, merge the data together into one object and then return. The issue I am having is I'm basically stuck in async/await hell and when pushing to a second array within the .then() block, the second array named
clone returns []. How can I make an api request, merge the data and return to the requester as needed?
Fetch code:
export default async function getProduct(product_id) {
const product = await fetch(
`${process.env.PRIVATE_APP_URL}/products/${product_id}.json`,
{
method: "GET",
headers: {
"Content-Type": "application/json",
},
}
).then((result) => {
return result.json();
});
return product.product;
}
API Handler:
const recharge_subscription_res = await rechargeAPI(
"GET",
`https://api.rechargeapps.com/subscriptions?customer_id=${recharge_customer.id}`
);
const closest_array = recharge_subscription_res.subscriptions.filter(
(e) => e.next_charge_scheduled_at == closest_date
);
let clone = [];
closest_array.forEach((element) => {
getProduct(element.shopify_product_id).then((product) => {
element.shopify_product_handle = product.handle;
element.shopify_product_image_url = product.image.src;
clone.push(element);
});
});
console.log(clone);
clone should log as an array of objects like closest_array, but instead logs as just an empty array. This isn't exactly like the other seemingly duplicate questions because typically their feature doesn't require sending the promise's data back to an external source. Most questions are related to the front end of things. My situation is with an Express.js API. Any help would be appreciated.

The original promise spec used .then(), and the new syntax hides then's with await. Style-wise, it makes sense to choose just one style and go with it.
In either style, there's a little challenge having to do with creating many promises in a loop. The js iteration functions (like map and forEach) take synchronous functions. The most common design is to create a collection of promises in a synchronous loop, then run them concurrently with Promise.all(). Taking both ideas into account...
You could (but don't have to) rewrite your network request like this...
// since we decorated "async" let's use await...
export default async function getProduct(product_id) {
const url = `${process.env.PRIVATE_APP_URL}/products/${product_id}.json`;
const options = { method: "GET", headers: { "Content-Type": "application/json" }};
const result = await fetch(url, options);
const product = await result.json();
return product.product;
}
await is not permitted at the top-level; it may be used only within an async function. Here, I'll make up a name and guess about the parameter
async function rechargeAndLookupProduct(recharge_customer) {
const base = 'https://api.rechargeapps.com/subscriptions';
const query = `customer_id=${recharge_customer.id}`;
const recharge_subscription_res = await rechargeAPI("GET",`${base}?${query}`);
const closest_array = recharge_subscription_res.subscriptions.filter(e =>
e.next_charge_scheduled_at == closest_date
);
// here's the important part: collect promises synchronously
// execute them together with Promise.all()
const promises = closest_array.map(element => {
return getProduct(element.shopify_product_id)
});
const allProducts = await Promise.all(promises);
// allProducts will be an array of objects that the promises resolved to
const clones = allProducts.map((product, i) => {
// use Object.assign so we'll really have a "clone"
let closest = Object.assign({}, closest_array[i]);
closest.shopify_product_handle = product.handle;
closest.shopify_product_image_url = product.image.src;
return closest;
});
// if I didn't make any typos (which I probably did), then
// clones ought to contain the result you expect
console.log(clones);
}

Your code has a flaw (in the section shown below). You have a dangling promise that you forgot to await or return.
When you log clone, none of the async getProduct operations have completed yet, and none of the elements have been pushed.
let clone = [];
closest_array.forEach((element) => {
getProduct(element.shopify_product_id).then((product) => {
element.shopify_product_handle = product.handle;
element.shopify_product_image_url = product.image.src;
clone.push(element);
}); // FLAW: dangling .then
});
console.log(clone); // FLAW: clone is not ready yet.
I would set it up more like this:
let clone = await Promise.all(closest_array.map((element) =>
getProduct(element.shopify_product_id).then((product) => {
element.shopify_product_handle = product.handle;
element.shopify_product_image_url = product.image.src;
return element;
})
));
console.log(clone);
It's a little sketchy to modify element the way you are (I wouldn't), but this way the getProduct calls are all in flight together for maximum efficiency. Promise.all handles awaiting all the promises and putting each's result into an array of results, which you can then await as a single promise since the calling function is async.

Am I using Promise.all correctly?

I am using it many places in my code and it works. but at one place it didn't give any error, also did not give me the desired result. and when I show my code to the support forum they suggest that "You are using the JS object/class “Promise” incorrectly."
Can Anyone guide me on what's wrong with my code
here is my code sample:
let charity = {};
await Promise.all(
charity = charityData.map(function( data ) {
let address = data.zipCode
let url = "https://maps.googleapis.com/maps/api/geocode/json?&address="+`'${address}'`+"&key=***Google geocoding Key***"; //client's Key
let urlResponse = Backendless.Request.get(url)
// let latitude = urlResponse.results[0].geometry.location.lat;
// let longitude = urlResponse.results[0].geometry.location.lng;
//let updateCharitiesData = {'objectId': data.objectId, 'latitude':latitude, 'longitude':longitude};
return urlResponse;
})
);
return charity;

Almost. Assuming Backendless.Request.[method] returns a promise it would be more correct to do something along the lines of:
async function getCharityData() {
const charity = await Promise.all(charityData.map( async function(data) {
const address = data.zipCode;
const url =
`https://maps.googleapis.com/maps/api/geocode/json?&address=${address}&key=***Google geocoding Key***`; //client's Key
const urlResponse = await Backendless.Request.get(url);
return urlResponse;
}));
return charity
}
Promise.all requires an array as its argument to work correctly; passing an Array.map here and assigning the returned value to charity both ensures your Promise.all runs as expected and the returned array is an array of resolved promises.

I would do it like this:
function getCharityData() {
// `charity` is an array of Promises that will each resolve to
// a response.
const charity = charityData.map((data) => {
let address = data.zipCode;
let url = 'https://maps.googleapis.com/maps/api/geocode'
let urlResponse = Backendless.Request.get(url);
return urlResponse;
});
return Promise.all(charity);
}
try {
const charityData = await getCharityData();
} catch (e) {
console.error(e);
}
This way, charityData will be an array of fetched responses.
In your code, the result of Promise.all() is never assigned to charity before it's returned, and that is the value you want.

If you have access to Async/Await I'd simply do the following:
function getCharityData(charityData) {
let results = [];
for (let i = 0; i < charityData.length; i++) {
let url = `https://maps.googleapis.com/maps/api/geocode/json?&address=${charityData[i].zipCode}&key=***Google geocoding Key***`;
try {
let result = await Backendless.Request.get(url);
results.push(result);
} catch (err) {
console.log("Oh dear!");
}
}
return results;
}
For your use case, there's no need to use any Promise libraries when you have Async/Await, good old fashioned for loops and await (I'd personally prefer to do this sort of call sequentially instead of in parallel like Promise.all implores when I'm querying external APIs. This also ensures we don't fail fast like Promise.all does.).

Array of filtered axios results from paginated API is empty

In my code below I get an empty array on my console.log(response) but the console.log(filterdIds) inside the getIds function is showing my desired data. I think my resolve is not right.
Note that I run do..while once for testing. The API is paged. If the records are from yesterday it will keep going, if not then the do..while is stopped.
Can somebody point me to the right direction?
const axios = require("axios");
function getToken() {
// Get the token
}
function getIds(jwt) {
return new Promise((resolve) => {
let pageNumber = 1;
const filterdIds = [];
const config = {
//Config stuff
};
do {
axios(config)
.then((response) => {
response.forEach(element => {
//Some logic, if true then:
filterdIds.push(element.id);
console.log(filterdIds);
});
})
.catch(error => {
console.log(error);
});
} while (pageNumber != 1)
resolve(filterdIds);
});
}
getToken()
.then(token => {
return token;
})
.then(jwt => {
return getIds(jwt);
})
.then(response => {
console.log(response);
})
.catch(error => {
console.log(error);
});
I'm also not sure where to put the reject inside the getIds function because of the do..while.

The fundamental problem is that resolve(filterdIds); runs synchronously before the requests fire, so it's guaranteed to be empty.
Promise.all or Promise.allSettled can help if you know how many pages you want up front (or if you're using a chunk size to make multiple requests--more on that later). These methods run in parallel. Here's a runnable proof-of-concept example:
const pages = 10; // some page value you're using to run your loop
axios
.get("https://httpbin.org") // some initial request like getToken
.then(response => // response has the token, ignored for simplicity
Promise.all(
Array(pages).fill().map((_, i) => // make an array of request promisess
axios.get(`https://jsonplaceholder.typicode.com/comments?postId=${i + 1}`)
)
)
)
.then(responses => {
// perform your filter/reduce on the response data
const results = responses.flatMap(response =>
response.data
.filter(e => e.id % 2 === 0) // some silly filter
.map(({id, name}) => ({id, name}))
);
// use the results
console.log(results);
})
.catch(err => console.error(err))
;
<script src="https://unpkg.com/axios/dist/axios.min.js"></script>
The network tab shows the requests happening in parallel:
If the number of pages is unknown and you intend to fire requests one at a time until your API informs you of the end of the pages, a sequential loop is slow but can be used. Async/await is cleaner for this strategy:
(async () => {
// like getToken; should handle err
const tokenStub = await axios.get("https://httpbin.org");
const results = [];
// page += 10 to make the snippet run faster; you'd probably use page++
for (let page = 1;; page += 10) {
try {
const url = `https://jsonplaceholder.typicode.com/comments?postId=${page}`;
const response = await axios.get(url);
// check whatever condition your API sends to tell you no more pages
if (response.data.length === 0) {
break;
}
for (const comment of response.data) {
if (comment.id % 2 === 0) { // some silly filter
const {name, id} = comment;
results.push({name, id});
}
}
}
catch (err) { // hit the end of the pages or some other error
break;
}
}
// use the results
console.log(results);
})();
<script src="https://unpkg.com/axios/dist/axios.min.js"></script>
Here's the sequential request waterfall:
A task queue or chunked loop can be used if you want to increase parallelization. A chunked loop would combine the two techniques to request n records at a time and check each result in the chunk for the termination condition. Here's a simple example that strips out the filtering operation, which is sort of incidental to the asynchronous request issue and can be done synchronously after the responses arrive:
(async () => {
const results = [];
const chunk = 5;
for (let page = 1;; page += chunk) {
try {
const responses = await Promise.all(
Array(chunk).fill().map((_, i) =>
axios.get(`https://jsonplaceholder.typicode.com/comments?postId=${page + i}`)
)
);
for (const response of responses) {
for (const comment of response.data) {
const {name, id} = comment;
results.push({name, id});
}
}
// check end condition
if (responses.some(e => e.data.length === 0)) {
break;
}
}
catch (err) {
break;
}
}
// use the results
console.log(results);
})();
<script src="https://unpkg.com/axios/dist/axios.min.js"></script>
(above image is an except of the 100 requests, but the chunk size of 5 at once is visible)
Note that these snippets are proofs-of-concept and could stand to be less indiscriminate with catching errors, ensure all throws are caught, etc. When breaking it into sub-functions, make sure to .then and await all promises in the caller--don't try to turn it into synchronous code.
See also
How do I return the response from an asynchronous call? and Why is my variable unaltered after I modify it inside of a function? - Asynchronous code reference which explain why the array is empty.
What is the explicit promise construction antipattern and how do I avoid it?, which warns against adding a new Promise to help resolve code that already returns promises.

To take a step back and think about why you ran into this issue, we have to think about how synchronous and asynchronous javascript code works together. Your
synchronous getIds function is going to run to completion, stepping through each line until it gets to the end.
The axios function invocation is returning a Promise, which is an object that represents some future fulfillment or rejection value. That Promise isn't going to resolve until the next cycle of the event loop (at the earliest), and your code is telling it to do some stuff when that pending value is returned (which is the callback in the .then() method).
But your main getIds function isn't going to wait around... it invokes the axios function, gives the Promise that is returned something to do in the future, and keeps going, moving past the do/while loop and onto the resolve method which returns a value from the Promise you created at the beginning of the function... but the axios Promise hasn't resolved by that point and therefore filterIds hasn't been populated.
When you moved the resolve method for the promise you're creating into the callback that the axios resolved Promise will invoke, it started working because now your Promise waits for axios to resolve before resolving itself.
Hopefully that sheds some light on what you can do to get your multi-page goal to work.
I couldn't help thinking there was a cleaner way to allow you to fetch multiple pages at once, and then recursively keep fetching if the last page indicated there were additional pages to fetch. You may still need to add some additional logic to filter out any pages that you batch fetch that don't meet whatever criteria you're looking for, but this should get you most of the way:
async function getIds(startingPage, pages) {
const pagePromises = Array(pages).fill(null).map((_, index) => {
const page = startingPage + index;
// set the page however you do it with axios query params
config.page = page;
return axios(config);
});
// get the last page you attempted, and if it doesn't meet whatever
// criteria you have to finish the query, submit another batch query
const lastPage = await pagePromises[pagePromises.length - 1];
// the result from getIds is an array of ids, so we recursively get the rest of the pages here
// and have a single level array of ids (or an empty array if there were no more pages to fetch)
const additionalIds = !lastPage.done ? [] : await getIds(startingPage + pages, pages);
// now we wait for all page queries to resolve and extract the ids
const resolvedPages = await Promise.all(pagePromises);
const resolvedIds = [].concat(...resolvedPages).map(elem => elem.id);
// and finally merge the ids fetched in this methods invocation, with any fetched recursively
return [...resolvedIds, ...additionalIds];
}

Data disappearing after async http calls in Angular

I have to code a tree component that displays multiple data, which worked fine with mocked data for me. The problem here is when I try to get data from servers, let me explain:
I have three main objects : Districts, buildings and doors. As you may guess, doors refers to buildingId and buildingID to districts. So to retrieve data and create my tree nodes, I have to do some http calls in forEach loops which is not asynchronous.
I won't share with you everything but just a minimized problem so I can get help easily:
This method retrieves a district array from the server and puts it in a local array:
async getDistricts(){
this.districtDataService.getDistrictData().toPromise().then(async districts => {
this.districts = await districts.results as District[];
});
}
On my ngOnInit :
ngOnInit() {
this.getDistricts().then(async () => {
console.log(this.districts);
for (const district of this.districts){
console.log(district);
}
})
The first console.log (in NgOnInit) returns an empty array, which is quite surprising because the first method puts the data in "this.districts". and logging data in the first method just after I put it in returns an array with my data. I guess it have something to do with the async/await I've used. Can anyone help?
EDIT 1: Tried to use this.getDistricts().finally() instead of this.getDistricts().then(), but didn't work.
EDIT 2: console.log in getDistrict get executed after the one before my loop. The expected behavior would be the opposite.
SOLVED: putting the for loop in a finally block after my HTTP call solves this. So as the answer says, I think I'm over engineering the async/await calls. I have to rethink my work based on this. Thank you everyone!

Well, you should return your Promise from getDistricts. Also you are very much over engineering and complicating the async/await concept. I understand you don't want to use Observables, but I would advise you to use them anyways.
With promises and async/await so you kinda see how to use them:
async getDistricts(): Promise<District[]> {
const { results } = await this.districtDataService.getDistrictData();
return results;
}
async ngOnInit(): Promise<void> {
this.districts = await this.getDistricts();
for (const district of this.districts){
console.log(district);
}
}
With Observable it would look like this:
getDistricts(): Observable<District[]> {
return this.districtDataService.getDistrictData().pipe(
map(({ results }) => results as District[])
);
}
ngOnInit(): void {
this.getDistricts().subscribe((districts) => {
this.districts = districts;
for (const district of this.districts){
console.log(district);
}
});
}

Just to provide whoever needs to make multiple http calls in a desired order.
as mentionned by others, i overcomplicated the concept of async await.
The trick was to use observables, convert them to Promises using .toPromise(), using .then() to get data into my variables, then making other async calls in finally block using .finally(async () => { ... }).
here's what my final code looks like :
async ngOnInit(): Promise<void> {
await this.districtDataService.getDistrictData().toPromise().then(response => {
this.districts = response.results as District[];
console.log(this.districts);
}).finally(async () => {
for (const district of this.districts){
await this.districtDataService.getBuildingsOfDistrict(district.id).toPromise().then(response => {
this.buildings = response.results as Building[];
console.log(this.buildings);
}).finally(async ()=> {
for(const building of this.buildings){
await this.districtDataService.getDoorsOfBuilding(building.id).toPromise().then(response => {
this.doors = response.results as Door[];
console.log(this.doors);
}).finally(async () => {
for(const door of this.doors){
await this.doorNodes.push(new districtNodeImpl(false,null,null,door,null));
}
})
await this.buildingNodes.push(new districtNodeImpl(false,null,building,null,this.doorNodes));
}
})
await this.dataSource.push(new districtNodeImpl(false,district,null,null,this.buildingNodes));
console.log(this.dataSource);
this.buildingNodes = new Array();
this.doorNodes = new Array();
}
})
Hope this will help ! have a nice day.