We Keep Coding

Changing a loop to Promise.all()

I'm currently iterating through an array, making an api request, waiting for it to resolve, then moving onto the next api call.
I've read of using Promise.all() and I think I could make these api calls in parallel and wait for that promise to resolve, but I'm not exactly sure how to translate what I have here to use Promise.all().
async lockFolder(folderModel) {
const driveId = folderModel.driveId;
// THIS IS WHAT I'D LIKE TO TRANSLATE TO USE Promise.all()
for (const file of folderModel.docs) {
let res = await this._lockFile(file.id, driveId);
}
// return something....
}
async _lockFile(fileId, driveId) {
try {
return await axios.post(myRequestOmittedForBrevity);
} catch (err) {
//some error
}
}
Is there a good way to translate my loop to Promise.all()? Can I still use await for it's response? Most examples I've seen use .then() but I've been trying to stick with await. Any help is appreciated!

It makes total sense to use Promise.all or Promise.allSettled for this, as soon your requests aren't dependent on each other.
It can be something like this:
async lockFolder(folderModel) {
const driveId = folderModel.driveId;
// THIS IS WHAT I'D LIKE TO TRANSLATE TO USE Promise.all()
const listOfPendingPromises = folderModel.docs.map(file => this._lockFile(file.id, driveId))
const resultArray = await Promise.all(listOfPendingPromises)
// return something....
}
async _lockFile(fileId, driveId) {
return await axios.post(myRequestOmittedForBrevity);
}

Array of filtered axios results from paginated API is empty

In my code below I get an empty array on my console.log(response) but the console.log(filterdIds) inside the getIds function is showing my desired data. I think my resolve is not right.
Note that I run do..while once for testing. The API is paged. If the records are from yesterday it will keep going, if not then the do..while is stopped.
Can somebody point me to the right direction?
const axios = require("axios");
function getToken() {
// Get the token
}
function getIds(jwt) {
return new Promise((resolve) => {
let pageNumber = 1;
const filterdIds = [];
const config = {
//Config stuff
};
do {
axios(config)
.then((response) => {
response.forEach(element => {
//Some logic, if true then:
filterdIds.push(element.id);
console.log(filterdIds);
});
})
.catch(error => {
console.log(error);
});
} while (pageNumber != 1)
resolve(filterdIds);
});
}
getToken()
.then(token => {
return token;
})
.then(jwt => {
return getIds(jwt);
})
.then(response => {
console.log(response);
})
.catch(error => {
console.log(error);
});
I'm also not sure where to put the reject inside the getIds function because of the do..while.

The fundamental problem is that resolve(filterdIds); runs synchronously before the requests fire, so it's guaranteed to be empty.
Promise.all or Promise.allSettled can help if you know how many pages you want up front (or if you're using a chunk size to make multiple requests--more on that later). These methods run in parallel. Here's a runnable proof-of-concept example:
const pages = 10; // some page value you're using to run your loop
axios
.get("https://httpbin.org") // some initial request like getToken
.then(response => // response has the token, ignored for simplicity
Promise.all(
Array(pages).fill().map((_, i) => // make an array of request promisess
axios.get(`https://jsonplaceholder.typicode.com/comments?postId=${i + 1}`)
)
)
)
.then(responses => {
// perform your filter/reduce on the response data
const results = responses.flatMap(response =>
response.data
.filter(e => e.id % 2 === 0) // some silly filter
.map(({id, name}) => ({id, name}))
);
// use the results
console.log(results);
})
.catch(err => console.error(err))
;
<script src="https://unpkg.com/axios/dist/axios.min.js"></script>
The network tab shows the requests happening in parallel:
If the number of pages is unknown and you intend to fire requests one at a time until your API informs you of the end of the pages, a sequential loop is slow but can be used. Async/await is cleaner for this strategy:
(async () => {
// like getToken; should handle err
const tokenStub = await axios.get("https://httpbin.org");
const results = [];
// page += 10 to make the snippet run faster; you'd probably use page++
for (let page = 1;; page += 10) {
try {
const url = `https://jsonplaceholder.typicode.com/comments?postId=${page}`;
const response = await axios.get(url);
// check whatever condition your API sends to tell you no more pages
if (response.data.length === 0) {
break;
}
for (const comment of response.data) {
if (comment.id % 2 === 0) { // some silly filter
const {name, id} = comment;
results.push({name, id});
}
}
}
catch (err) { // hit the end of the pages or some other error
break;
}
}
// use the results
console.log(results);
})();
<script src="https://unpkg.com/axios/dist/axios.min.js"></script>
Here's the sequential request waterfall:
A task queue or chunked loop can be used if you want to increase parallelization. A chunked loop would combine the two techniques to request n records at a time and check each result in the chunk for the termination condition. Here's a simple example that strips out the filtering operation, which is sort of incidental to the asynchronous request issue and can be done synchronously after the responses arrive:
(async () => {
const results = [];
const chunk = 5;
for (let page = 1;; page += chunk) {
try {
const responses = await Promise.all(
Array(chunk).fill().map((_, i) =>
axios.get(`https://jsonplaceholder.typicode.com/comments?postId=${page + i}`)
)
);
for (const response of responses) {
for (const comment of response.data) {
const {name, id} = comment;
results.push({name, id});
}
}
// check end condition
if (responses.some(e => e.data.length === 0)) {
break;
}
}
catch (err) {
break;
}
}
// use the results
console.log(results);
})();
<script src="https://unpkg.com/axios/dist/axios.min.js"></script>
(above image is an except of the 100 requests, but the chunk size of 5 at once is visible)
Note that these snippets are proofs-of-concept and could stand to be less indiscriminate with catching errors, ensure all throws are caught, etc. When breaking it into sub-functions, make sure to .then and await all promises in the caller--don't try to turn it into synchronous code.
See also
How do I return the response from an asynchronous call? and Why is my variable unaltered after I modify it inside of a function? - Asynchronous code reference which explain why the array is empty.
What is the explicit promise construction antipattern and how do I avoid it?, which warns against adding a new Promise to help resolve code that already returns promises.

To take a step back and think about why you ran into this issue, we have to think about how synchronous and asynchronous javascript code works together. Your
synchronous getIds function is going to run to completion, stepping through each line until it gets to the end.
The axios function invocation is returning a Promise, which is an object that represents some future fulfillment or rejection value. That Promise isn't going to resolve until the next cycle of the event loop (at the earliest), and your code is telling it to do some stuff when that pending value is returned (which is the callback in the .then() method).
But your main getIds function isn't going to wait around... it invokes the axios function, gives the Promise that is returned something to do in the future, and keeps going, moving past the do/while loop and onto the resolve method which returns a value from the Promise you created at the beginning of the function... but the axios Promise hasn't resolved by that point and therefore filterIds hasn't been populated.
When you moved the resolve method for the promise you're creating into the callback that the axios resolved Promise will invoke, it started working because now your Promise waits for axios to resolve before resolving itself.
Hopefully that sheds some light on what you can do to get your multi-page goal to work.
I couldn't help thinking there was a cleaner way to allow you to fetch multiple pages at once, and then recursively keep fetching if the last page indicated there were additional pages to fetch. You may still need to add some additional logic to filter out any pages that you batch fetch that don't meet whatever criteria you're looking for, but this should get you most of the way:
async function getIds(startingPage, pages) {
const pagePromises = Array(pages).fill(null).map((_, index) => {
const page = startingPage + index;
// set the page however you do it with axios query params
config.page = page;
return axios(config);
});
// get the last page you attempted, and if it doesn't meet whatever
// criteria you have to finish the query, submit another batch query
const lastPage = await pagePromises[pagePromises.length - 1];
// the result from getIds is an array of ids, so we recursively get the rest of the pages here
// and have a single level array of ids (or an empty array if there were no more pages to fetch)
const additionalIds = !lastPage.done ? [] : await getIds(startingPage + pages, pages);
// now we wait for all page queries to resolve and extract the ids
const resolvedPages = await Promise.all(pagePromises);
const resolvedIds = [].concat(...resolvedPages).map(elem => elem.id);
// and finally merge the ids fetched in this methods invocation, with any fetched recursively
return [...resolvedIds, ...additionalIds];
}

java script promises lingo

I'm new to promises, so apologize for the newbie question. Inside my function, I have the following lines (middle of the function):
...
const retPromise = await buildImgs(file, imgArray);
retPromise.then(async function () {
console.log("completed build imgs");
...
My assumption was that the "then" from the promise would not execute until the await was completed. but alas, it is acting like sync code, and the retPromise evaluating the "then" before the buildImgs is completed (as measured by my console.log flows). The result is an undefined retPromise.
please help...what am I missing in the concept?
OK: after feedback, let me explaing further my question:
I am trying to understand this async/sync flow and control concept:
const retVal = somefunc();
console.log(retVal);
const retVal = await somefunc();
console.log(retVal);
in the first case, if I understand sync / async code correctly, then I should have a possibility that retVal is undefined when the console.log finds it...
in the second case, I thought it would stop flow until that point that somefunc() completes, then the flow would continue. However my reading seems to indicate it will still try to run the console.log message as a parallel thread. So this leads me to believe I would need to put the console.log inside of the .then after somefunc. Which leads me to promises. So I made a promise return, which I see happening.
However, the .then, as in my original post code, seems to post the console message "completed build imgs", before code inside my buildImgs completes (measured by time I know the function to take, and also console messages inside the buildImgs to help me with sequencing)
so it seems to me I am still missing a fundamental on how to block flow for async code. :(

When you use await construction the script waits until the promise resolves and return to your retPromise value from this promise.
So in this case better to choose one. Remove await and keep then, or keep await and use retPromise value.

Assuming that buildImgs is actually returning a promise (example)
const buildImgs = (file, imgArray) => {
return new Promise((resolve, reject) => {
try {
// ...
resolve()
} catch (err) {
reject(err)
}
})
}
When you call await on a promise its already waiting for the promise to complete, if you remove the await on the call then you can use .then
there are two ways to write promise handlers, the older way looks like this
buildImgs(file, imgArray)
.then(() => {
console.log('im done')
})
.catch(err => {
console.error(err)
})
and the newer syntax
// you must declare "async" in order to use "await"
const asyncFunction = async () => {
try {
await buildImgs(file, imgArray)
console.log('im done')
} catch(err) {
console.error(err)
}
}
asyncFunction()
if you need the return value from the promise, then you would assign it to a variable
const ineedResponse = await buildImgs(file, imgArray)
console.log(ineedResponse)
// or
buildImgs(file, imgArray)
.then(ineedResponse => {
console.log(ineedResponse)
})
.catch(err => {
console.error(err)
})

NodeJs async/await again

I just can't get my head around this stuff :(
compatibleApps: async () => {
common.header('Install Compatible Apps')
const compatibleApps = JSON.parse(fs.readFileSync('./data/compatibleApps.json', 'utf8'));
const value = await inquirer.compatibleApps();
for (let element of value.removeAppsList) {
for (let element2 of compatibleApps) {
if (element === element2.name) {
await files.downloadFile(element2)
}
}
}
await adb.installApk()
},
await adb.installApk() is being executed before the all calls of await files.downloadFile(element2 ) have been completed..
Below is the contents of downloadFile, I guess I need to wrap it in a promise?
downloadFile: async (element) => {
option = {
dir: './data/apps',
onDone: (info)=>{
console.log('Latest ' + element.name + ' Downloaded')
},
onError: (err) => {
console.log('error', err);
},
onProgress: (curr, total) => {
},
}
var dd = await dl(element.url, option);
}

Keep in mind that await only does anything useful if you are awaiting a promise that is linked to your actual asynchronous operation. "Linked" in that sentence means that the promise resolves when the asynchronous operation is done or rejected if it has an error.
If the function you are awaiting either returns nothing or returns just a plain value, yet contains asynchronous operations, then the await doesn't actually await anything. It calls the function, initiates those asynchronous operations, the function returns a non-promise value or returns an already resolved promise, the await doesn't have a anything to wait for and just continues executing more lines of code without the expected pause for the asynchronous operations to complete.
So, in your code, the only way that:
await adb.installApk()
executes before any of the calls to:
await files.downloadFile(element2)
is if files.downloadFile() does not actually return a promise that is linked to the asynchronous operations it contains or perhaps if you never execute files.downloadFile(element2) because of the conditionals.
For more specific help, show us the code for files.downloadFile() and confirm that you are getting through your conditionals and executing it at least once.

Asynchonous issue with loop in nodejs

I'm trying to iterate threw a list of item and do some actions on them by calling an API like this example :
for (i = 0; i < arr.length; i++) {
if (arr[i].id == 42) {
api.requestAction(arr[i].id, function(error, response){ });
}
}
Problem is the loop obviously ended before all the requests are done and the program exits. What should I do to manage it ? I saw the "Promise" method but don't really know how I can use it in this case or maybe there's an other solution.
Thank you by advance !

With node-fetch (a promisify http api) you can together with async/await halt the for loop until it's done but this requires node v6+ with --harmony-async-await flag added
const fetch = require('node-fetch')
async function foo() {
for (let item of arr) {
if (item.id == 42) {
let res = await fetch(url)
let body = await res.text()
console.log(body)
}
}
console.log('done (after request)')
}
now every time you add the async keyword in front of a function it will always return a promise that resolve/rejects when everything is done
foo().then(done, fail)
alternetive you can just wrap you api fn in a promise if you don't want to install node-fetch
await new Promise((rs, rj) => {
api.requestAction(arr[i].id, function(error, response){
error ? rj(error) : rs(response)
})
})

Install bluebird
npm install bluebird --save
Code
//require npm
var Promise = require("bluebird");
//code
//"promisify" converts traditional callback function into a Promise based function
var _requestAction = Promise.promisify(api.requestAction);
//loop over array
Promise.map(arr, function (value) {
if (value.id == 42) {
//async request
return _requestAction(value.id).then(function (_result) {
//success
console.log(_result);
}).catch(function (e) {
//error
console.error(e);
});
}
});

You could use async.js. It's an asyncronous control flow library which provides control flows for things like sequential loops, looping in parralel, and many other common flow control mechanism, check it out.
See code below, the code assumes that you're variable 'arr' is defined somewhere in scope.
npm install async
var async = require("async");
//Loop through each item, waiting for your
//asyncronous function to finish before continuing
//to move onto the next item in the array
//NOTE: This does not loop sequentially, if you want that function with asyncjs then user eachSeries
async.each(arr,
//Item is the current item being iterated over,
//callback is the callback you call to finish the current iteration, it accepts an error and result parameter callback(error, result);
function (item, callback) {
api.requestAction(item.id, function(error, response){
//Check for any errors...
if (error) return callback(error);
callback(null);
});
},
function (err, result) {
//You've now finished the loop
if (err) {
//Do something, you passed an error object to
//in one of the loop's iterations
}
//No errors, move on with your code..
});

Use Bluebird Promises:
var Promise = require('bluebird');
Promise.map(arrayOfIds, function(item){
return api.requestAction(item);
})
.then(function(response){
// all the requests are resolved here
})
if u want sequential execution of the ids then use Promise.mapSeries (is slow as it waits for task to finish)