We Keep Coding

NodeJS: Chain functions automatically in a promise?

I'm currently fetching data from an API and I need to do multiple GET requests (using axios). After all those GET requests are completed, I return a resolved promise.
However, I need to do these GET requests automatically based on an array list:
function do_api_get_requests() {
return promise = new Promise(function(resolve, reject) {
API_IDs = [0, 1, 2];
axios.get('https://my.api.com/' + API_IDs[0])
.then(data => {
// Do something with data
axios.get('https://my.api.com/' + API_IDs[1])
.then(data => {
// Do something with data
axios.get('https://my.api.com/' + API_IDs[2])
.then(data => {
// Do something with data
// Finished, resolve
resolve("success");
}
}
}
}
}
This works but the problem is API_IDs isn't always going to be the same array, it will change. So I'm not sure how to chain these requests automatically.

Since you said it may be a variable length array and you show sequencing the requests, you can just loop through the array using async/await:
async function do_api_get_requests(API_IDS) {
for (let id of API_IDS) {
const data = await axios.get(`https://my.api.com/${id}`);
// do something with data here
}
return "success";
}
And, since you said the list of API ids would be variable, I made it a parameter that you can pass into the function.
If you wanted to run all the API requests in parallel (which might be OK for a small array, but might be trouble for a large array) and you don't need to run them in a specific order, you can do this:
function do_api_get_requests(API_IDS) {
return Promise.all(API_IDS.map(async (id) => {
const data = await axios.get(`https://my.api.com/${id}`);
// do something with data here for this request
})).then(() => {
// make resolved value be "success"
return "success";
});
}
Depending upon your circumstances, you could also use Promise.allSettled(). Since you don't show getting results back, it's not clear whether that would be useful or not.

You can use Promise.all() method to do all API requests at the same time, and resolve when all of them resolves.
function do_api_get_requests() {
const API_IDs = [0, 1, 2];
let promises = [];
for (const id of API_IDS) {
promises.push(axios.get(`https://my.api.com/${id}`));
}
return Promise.all(promises);
}

If you use Bluebird.js (a better promise library, and faster than the in-built Promise), you can use Promise.each(), Promise.mapSeries(), or Promisme.reduce() to do what you want.
http://bluebirdjs.com

promises.push() runs instantly and does not wait for promises.all()

I have a nodejs function processReviews(workflow) that when called, is supposed to push multiple promises to an array promises[], then run them all with promises.all() after the for loop.
function examplePromiseFunc(){
return new Promise((resolve, reject) => {
console.log("examplePromiseFunc() INSIDE ")
resolve('done')
})
}
async function processReviews(workflow){
//get objects from s3
let allObjects = await getAllObjects(workflow);
allObjects = allObjects.filter(obj => obj.Key.includes('output.json'))
console.log(`found ${allObjects.length} .json files.`)
const promises = [];
for (let i = 0; i < allObjects.length; i++) {
console.log('i=',i,' pushing to promises[]')
promises.push( examplePromiseFunc() )
}
const result = await Promise.all(promises)
console.log('running, result = ', result);
}
But when I run my code, the output looks like this:
found 697 .json files.
i= 0 pushing to promises[]
examplePromiseFunc() INSIDE
i= 1 pushing to promises[]
examplePromiseFunc() INSIDE
i= 2 pushing to promises[]
examplePromiseFunc() INSIDE
i= 3 pushing to promises[]
examplePromiseFunc() INSIDE
...
Which means that each time I push a promise to my promises[] array (promises.push( await examplePromiseFunc() )) the function examplePromiseFunc() is getting called instantly and is not waiting.
I want my function to only get called when I run await Promise.all(promises) at the end, is there something I'm missing? are my async functions causing a problem? I've been reading up on javascript promises.all and this seems like a fine implementation.

The issue is that you are already using await inside your loop, which means the loop will "wait" and process the items sequentially.
Instead, you should just add the promises to the array, then await all of them at the end like you had:
async function processReviews(workflow) {
//get objects from s3
const allObjects = await getAllObjects(workflow);
const promises = [];
for (let i = 0; i < allObjects.length; i++) {
// Don't await the promise here, just start it and add it to the array.
promises.push(examplePromiseFunc(allObjects[i]));
}
const result = await Promise.all(promises)
console.log(result);
}

There is a fundamental misunderstanding here of of how the Promise constructor works.
The constructor takes a single function argument known as the executor:
new Promise( executor)
The executor function is called synchronously during construction with two function arguments commonly called resolve and reject:
executor( resolve, reject)
The promise executor is responsible for initiating an asynchronous operation (usually) and make the resolve and reject functions available to code that handles operation completion and error handling, often within a call back function.
Hence the code
for (let i = 0; i < allObjects.length; i++) {
console.log('i=',i,' pushing to promises[]')
promises.push( examplePromiseFunc() )
}
calls examplePromiseFunct multiple times and within that function, the executor for the promise returned is called synchronously (by Promise) during construction. Hence the log is as one would expect: a log of "examplePromiseFunc() INSIDE" each time examplePromiseFunc is called.
This misunderstanding probably leads to the second one:
Promise.all does not "run" promises - promises are passive objects that respond in deterministic ways to calling their associated resolve and reject functions, by calling either fulfilled or rejected handlers attached to them - or chaining off another promise if resolved with a promise.
Promise.all simply returns a promise that is fulfilled with an array of the fulfilled results of its argument promises, or is rejected with the rejection reason or the first promise in its argument array that becomes rejected. It has an executor in native code that effectively attaches then handlers to promises in its argument array and then passively waits (i.e. it returns to the event loop) until argument promises are settled one at a time.

that's because promises work this way only. When you are pushing a promise into an array you are waiting for it already(inside the loop) i.e if you would not wait for them to execute then also they would execute, with or without await promise.all, Also it is possible that all the promises already resolved before you pass that array in promise.all.
The below function would also resolve all promises without the promise all.
async function processReviews(workflow){
//get objects from s3
let allObjects = await getAllObjects(workflow);
allObjects = allObjects.filter(obj => obj.Key.includes('output.json'))
console.log(`found ${allObjects.length} .json files.`)
const promises = [];
for (let i = 0; i < allObjects.length; i++) {
console.log('i=',i,' pushing to promises[]')
promises.push( examplePromiseFunc() )
}
}
Also, you should not use promise.all without limit as it might reach limitations of your hardware.
Using a map with a limit can reduce your problem.
async function processReviews(workflow) {
//get objects from s3
let allObjects = await getAllObjects(workflow);
allObjects = allObjects.filter(obj => obj.Key.includes("output.json"));
console.log(`found ${allObjects.length} .json files.`);
for (let i = 0; i < allObjects.length; i = i + PROMISE_LIMIT) {
const objects = allObjects.slice(i, i + PROMISE_LIMIT);
const result = await Promise.all(objects.map(elem => examplePromiseFunc()));
console.log("running, result = ", result);
}
}

Function similar to Promise.some/any for an unknown amount of promises

I am creating a script in node.js (V8.1.3) which looks at similar JSON data from multiple API's and compares the values. To be more exact I am looking at different market prices of different stocks (actually cryptocurrencies).
Currently, I am using promise.all to wait for all responses from the respective APIs.
let fetchedJSON =
await Promise.all([getJSON(settings1), getJSON(settings2), getJSON(settings3) ... ]);
However, Promise.all throws an error if even just one promise rejects with an error. In the bluebird docos there is a function called Promise.some which is almost what I want. As I understand it takes an array of promises and resolves the two fastest promises to resolve, or otherwise (if less than 2 promises resolve) throws an error.
The problem with this is that firstly, I don't want the fastest two promises resolved to be what it returns, I want any successful promises to be returned, as long as there is more than 2. This seems to be what Promise.any does except with a min count of 1. (I require a minimum count of 2)
Secondly, I don't know how many Promises I will be awaiting on (In other words, I don't know how many API's I will be requesting data from). It may only be 2 or it may be 30. This depends on user input.
Currently writing this it seems to me there is probably just a way to have a promise.any with a count of 2 and that would be the easiest solution. Is this possible?
Btw, not sure if the title really summarizes the question. Please suggest an edit for the title :)
EDIT: Another way I may be writing the script is that the first two APIs to get loaded in start getting computed and pushed to the browser and then every next JSON that gets loaded and computed after it. This way I am not waiting for all Promises to be fulfilled before I start computing the data and passing results to the front end. Would this be possible with a function which also works for the other circumstances?
What I mean kind of looks like this:
Requesting JSON in parallel...
|-----JSON1------|
|---JSON-FAILS---| > catch error > do something with error. Doesn't effect next results.
|-------JSON2-------| > Meets minimum of 2 results > computes JSON > to browser.
|-------JSON3---------| > computes JSON > to browser.

How about thening all the promises so none fail, pass that to Promise.all, and filter the successful results in a final .then.
Something like this:
function some( promises, count = 1 ){
const wrapped = promises.map( promise => promise.then(value => ({ success: true, value }), () => ({ success: false })) );
return Promise.all( wrapped ).then(function(results){
const successful = results.filter(result => result.success);
if( successful.length < count )
throw new Error("Only " + successful.length + " resolved.")
return successful.map(result => result.value);
});
}

This might be somewhat clunky, considering you're asking to implement an anti-pattern, but you can force each promise to resolve:
async function fetchAllJSON(settingsArray) {
let fetchedJSON = await Promise.all(
settingsArray.map((settings) => {
// force rejected ajax to always resolve
return getJSON(settings).then((data) => {
// initial processing
return { success: true, data }
}).catch((error) => {
// error handling
return { success, false, error }
})
})
).then((unfilteredArray) => {
// only keep successful promises
return dataArray.filter(({ success }) => success)
})
// do the rest of your processing here
// with fetchedJSON containing array of data
}

You can use Promise.allSettled([]). the difference is that allSettled will return an array of objects after all the promises are settled regardless if successful or failed. then just find the successful o whatever you need.
let resArr = await Promise.allSettled(userNamesArr.map(user=>this.authenticateUserPassword(user,password)));
return resArr.find(e=>e.status!="rejected");
OR return resArr.find(e=>e.status=="fulfilled").

The other answers have the downside of having to wait for all the promises to resolve, whereas ideally .some would return as soon as any (N) promise(s) passes the predicate.
let anyNPromises = (promises, predicate = a => a, n = 1) => new Promise(async resolve => {
promises.forEach(async p => predicate(await p) && !--n && resolve(true));
await Promise.all(promises);
resolve(false);
});
let atLeast2NumbersGreaterThan5 = promises => anyNPromises(promises, a => a > 5, 2);
atLeast2NumbersGreaterThan5([
Promise.resolve(5),
Promise.resolve(3),
Promise.resolve(10),
Promise.resolve(11)]
).then(a => console.log('5, 3, 10, 11', a)); // true
atLeast2NumbersGreaterThan5([
Promise.resolve(5),
Promise.resolve(3),
Promise.resolve(10),
Promise.resolve(-43)]
).then(a => console.log('5, 3, 10, -43', a)); // false
atLeast2NumbersGreaterThan5([
Promise.resolve(5),
Promise.resolve(3),
new Promise(() => 'never resolved'),
Promise.resolve(10),
Promise.resolve(11)]
).then(a => console.log('5, 3, unresolved, 10, 11', a)); // true

How to create a loop of promises

so i have a promise that collects data from a server but only collects 50 responses at a time. i have 250 responses to collect.
i could just concate promises together like below
new Promise((resolve, reject) => {
resolve(getResults.get())
})
.then((results) => {
totalResults.concat(results)
return getResults.get()
})
.then((results) => {
totalResults.concat(results)
return getResults.get()
}).then((results) => {
totalResults.concat(results)
return getResults.get()
})
In this instance i only need 250 results so this seems a managable solution but is there a way of concating promises in a loop. so i run a loop 5 times and each time run the next promise.
Sorry i am new to promises and if this were callbacks this is what i would do.

If you want to loop and serialise the promises, not executing any other get calls once one fails, then try this loop:
async function getAllResults() { // returns a promise for 250 results
let totalResults = [];
try {
for (let i = 0; i < 5; i++) {
totalResults.push(...await getResults.get());
}
} catch(e) {};
return totalResults;
}
This uses the EcmaScript2017 async and await syntax. When not available, chain the promises with then:
function getAllResults() {
let totalResults = [];
let prom = Promise.resolve([]);
for (let i = 0; i < 5; i++) {
prom = prom.then(results => {
totalResults = totalResults.concat(results);
return getResults.get();
});
}
return prom.then(results => totalResults.concat(results));
}
Note that you should avoid the promise construction anti-pattern. It is not necessary to use new Promise here.
Also consider adding a .catch() call on the promise returned by the above function, to deal with error conditions.
Finally, be aware that concat does not modify the array you call it on. It returns the concatenated array, so you need to assign that return value. In your code you don't assign the return value, so the call has no effect.
See also JavaScript ES6 promise for loop.

Probably you just need Promise.all method.
For every request you should create a promise and put it in an array, then you wrap everything in all method and you're done.
Example (assuming that getResults.get returns a promise):
let promiseChain = [];
for(let i = 0; i <5; i++){
promiseChain.push(getResults.get());
}
Promise.all(promiseChain)
.then(callback)
You can read more about this method here:
Promise.all at MDN
EDIT
You can access data returned by the promises this way:
function callback(data){
doSomething(data[0]) //data from the first promise in the chain
...
doEventuallySomethingElse(data[4]) //data from the last promise
}

generators + promises to parallelize N number of items

the challenge:
We want to make N parallel ajax requests for an item's children.
Upon returning, we want to process them in sequential order (1...N)
We do NOT want to wait for all promises to return, but we want to process them IN ORDER as they come back.
For example:
Even if 2,3,5 come back before 1, we should hold onto the results of 2,3,5, and upon 1's return, process 1,2,3 in order (and wait for 4 to come back before 5)
Tools: Q + ES6 generators
Create array of N-1 length with placeholder variables
EG when N = 3:
let [N1,N2,N3] = yield [ Promise1, Promise2, Promise3 ]
//process items sequentially:
console.log(N1)
console.log(N2)
console.log(N3)
However, populating an array of empty variables doesn't seem to work of course because the reference doesn't know where to find the var declaration
for(var i = 0; i< 3; i++) {
res.push("some empty var")
}
Given the constraints of sticking to the tools provided, how could we parallelize calls, but process their returns sequentially?

You can use Promise.all(), .then()
javascript at Answer returns exact results described at Question
We want to make N parallel ajax requests for an item's children.
Upon returning, we want to process them in sequential order (1...N)
We do NOT want to wait for all promises to return, but we want to process them IN ORDER as they come back.
how could we parallelize calls, but process their returns
sequentially?
You can use .then() chained to original function which returns a promise or Promise object itself to process promise before returning value to be processed in sequential order of parameters passed to Promise.all() at .then() chained to Promise.all()
var n = 0;
var fn = function() {
return new Promise(function(resolve, reject) {
console.log("promise " + ++n + " called");
setTimeout(function(i) {
resolve(i)
}, Math.random() * 2500, n)
})
// handle requirement 3. here
.then(function(res) {
console.log(res);
return res
})
}
Promise.all([fn(), fn(), fn()]) // handle requirement 1. here
// handle requirement 2. here
.then(function(data) {
let [N1, N2, N3] = data;
console.log(N1, N2, N3);
})

You can do that by waiting for the next promise inside the loop:
const promises = […]; // or created programmatically
for (const promise of promises) {
const result = yield promise; // await them sequentially
console.log(result);
}