We Keep Coding

NodeJS: Chain functions automatically in a promise?

I'm currently fetching data from an API and I need to do multiple GET requests (using axios). After all those GET requests are completed, I return a resolved promise.
However, I need to do these GET requests automatically based on an array list:
function do_api_get_requests() {
return promise = new Promise(function(resolve, reject) {
API_IDs = [0, 1, 2];
axios.get('https://my.api.com/' + API_IDs[0])
.then(data => {
// Do something with data
axios.get('https://my.api.com/' + API_IDs[1])
.then(data => {
// Do something with data
axios.get('https://my.api.com/' + API_IDs[2])
.then(data => {
// Do something with data
// Finished, resolve
resolve("success");
}
}
}
}
}
This works but the problem is API_IDs isn't always going to be the same array, it will change. So I'm not sure how to chain these requests automatically.

Since you said it may be a variable length array and you show sequencing the requests, you can just loop through the array using async/await:
async function do_api_get_requests(API_IDS) {
for (let id of API_IDS) {
const data = await axios.get(`https://my.api.com/${id}`);
// do something with data here
}
return "success";
}
And, since you said the list of API ids would be variable, I made it a parameter that you can pass into the function.
If you wanted to run all the API requests in parallel (which might be OK for a small array, but might be trouble for a large array) and you don't need to run them in a specific order, you can do this:
function do_api_get_requests(API_IDS) {
return Promise.all(API_IDS.map(async (id) => {
const data = await axios.get(`https://my.api.com/${id}`);
// do something with data here for this request
})).then(() => {
// make resolved value be "success"
return "success";
});
}
Depending upon your circumstances, you could also use Promise.allSettled(). Since you don't show getting results back, it's not clear whether that would be useful or not.

You can use Promise.all() method to do all API requests at the same time, and resolve when all of them resolves.
function do_api_get_requests() {
const API_IDs = [0, 1, 2];
let promises = [];
for (const id of API_IDS) {
promises.push(axios.get(`https://my.api.com/${id}`));
}
return Promise.all(promises);
}

If you use Bluebird.js (a better promise library, and faster than the in-built Promise), you can use Promise.each(), Promise.mapSeries(), or Promisme.reduce() to do what you want.
http://bluebirdjs.com

Understanding Promise.all in javascript

I have a sample code and questions along with that
'use strict';
const AWS = require('aws-sdk');
const promiseFunction = (item) => {
const database = new AWS.DynamoDB();
const params = {
TableName: 'some-table',
Item: item
}
try {
//database.putItem(params).promise(); //3
//Updated as per the comments..
return database.putItem(params).promise(); //3
} catch (err) {
console.log(`Error`);
}
}
const test = async () => {
const arr = [];
try {
for (let data=1;data<=1000;data++) {
const obj = { data: {N: data.toString()}}
arr.push(promiseFunction(obj)); //1
}
const result = await Promise.all(arr); //2
} catch (e) {
console.log(`Error`);
}
}
test();
Some follow-up questions:
At line-2, the result will contain the resolve/reject result once all the promise based function get executed on line //1. Right?
How, the promiseFunction at line-1 is executing and inserting item in dynamodb as I am just pushing it into an array rather than calling the dynamo putItem API. Technically, all 'putItem' should start execute in parallel at line-2. Please help here?
Is the above piece of code is valid in terms of Promise.all and executing insert operations in parallel. Please advice.

At line-2, the result will contain the resolve/reject result once all the promise based function get executed on line //1. Right?
The result will be assigned once all the promises in the arr have been fulfilled. Promise.all doesn't know (or care) how those promises were created.
Technically, all 'putItem' should start execute in parallel at line-2.
No. All the putItem() calls are executed in your loop. Promise.all doesn't execute anything, it only waits for promises created earlier. The request are done concurrently because the code starts them all at once, without waiting in between each loop iteration, not because it uses Promise.all.
Is the above piece of code is valid in terms of Promise.all and executing insert operations in parallel.
No. Your promiseFunction doesn't actually return a promise, since you forgot a return statement on the line marked as //3. Once you add that, the code is fine though.

NodeJS Asynchronous

I am looking for ideas/help to improve my code. It's already working, but I am not confident with it and not really proud of it. This is short version of my function -
module.exports.serverlist = async () => {
let promises = [];
const serverlist = [];
serverlist.push({ mon_sid: 'AAA', mon_hostname: 'aaaa.com', mon_port: 80 })
serverlist.push({ mon_sid: 'BBB', mon_hostname: 'bbbb.com', mon_port: 80 })
serverlist.forEach(async (Server) => {
if (Server.mon_sid.includes('_DB')) {
// Function home.checkOracleDatabase return promise, same as above functions
promises.push(home.checkOracleDatabase(Server.mon_hostname, Server.mon_port));
} else if (Server.mon_sid.includes('_HDB')) {
promises.push(home.checkHANADatabase(Server.mon_hostname, Server.mon_port));
} else {
promises.push(home.checkPort(Server.mon_port, Server.mon_hostname, 1000));
}
})
for (let i = 0; i < serverlist.length; i++) {
serverlist[i].status = await promises[i];
}
console.table(serverlist);
What does the code do?:
- It asynchronously performing needed availability check of the service.
What is the expectation?
- That the code will run asynchronously, at the end of function it will wait for all promises to be resolved, for failed results it will perform the check over again, and to have more control over the promises. At this moment the promises are not really connected somehow with the array of systems, it just base on the order but it can be problematic in future.
If someone can assist/give some advises, I would be more than happy.
Also I am not sure how many parallel asynchronous operations NodeJS can perform (or the OS). Currently there are 30 systems on the list, but in future it can be 200-300, I am not sure if it can be handled at once.
Update
I used promise.all - it works fine. However the problem is as I mentioned, I don't have any control over the $promises array. The results as saved in the sequence as they were triggered, and this is how they are being assigned back to serverlist[i].status array from $promises. But I would like to have more control over it, I want to have some index, or something in the $promises array so I can make sure that the results are assigned to PROPER systems (not luckily counting that by the sequence it will be assigned as it should).
Also I would like to extend this function with option to reAttempt failed checks, and for that definitely I need some index in $promises array.
Update 2
After all of your suggestion this is how the code looks for now -
function performChecks(serverlist) {
const Checks = serverlist.map(Server => {
if (Server.mon_sid.includes('_DB')) {
let DB_SID = Server.mon_sid.replace('_DB', '');
return home.checkOracleDatabase(DB_SID, Server.mon_hostname, Server.mon_port)
} else if (Server.mon_sid.includes('_HDB')) {
let DB_SID = Server.mon_sid.replace('_HDB', '');
return home.checkHANADatabase(DB_SID, Server.mon_hostname, Server.mon_port);
} else {
return home.checkPort(Server.mon_port, Server.mon_hostname, 1000)
}
})
return Promise.allSettled(Checks)
}
// Ignore the way that the function is created, it's just for debug purpose
(async function () {
let checkResults = [];
let reAttempt = [];
let reAttemptResults = [];
const serverlist = [];
serverlist.push({ mon_id: 1, mon_sid: 'AAA', mon_hostname: 'hostname_1', mon_port: 3203 })
serverlist.push({ mon_id: 2, mon_sid: 'BBB', mon_hostname: 'hostname_2', mon_port: 3201 })
serverlist.push({ mon_id: 3, mon_sid: 'CCC', mon_hostname: 'hostname_3', mon_port: 3203 })
// Perform first check for all servers
checkResults = await performChecks(serverlist);
// Combine results from check into serverlist array under status key
for(let i = 0; i < serverlist.length; i++) {
serverlist[i]['status'] = checkResults[i].value;
}
// Check for failed results and save them under reAttempt variable
reAttempt = serverlist.filter(Server => Server.status == false);
// Perform checks again for failed results to make sure that it wasn't temporary netowrk/script issue/lag
// Additionally performChecks function will accept one more argument in future which will activate additional trace for reAttempt
reAttemptResults = await performChecks(reAttempt);
// Combine results from reAttempt checks into reAttempt array
for(let i = 0; i < reAttempt.length; i++) {
reAttempt[i]['status'] = reAttemptResults[i].value;
}
// Combine reAttempt array with serverlist array so serverlist can have latest updated data
serverlist.map(x => Object.assign(x, reAttempt.find(y => y.mon_id == x.mon_id)));
// View the results
console.table(serverlist);
})();

Firstly instead of doing a for each and push promises you can map them and do a Promise all. You need no push. Your function can return directly your promise all call. The caller can await it or use then...
Something like this (I didn't test it)
// serverlist declaration
function getList(serverlist) {
const operations = serverlist.map(Server => {
if (Server.mon_sid.includes('_DB')) {
return home.checkOracleDatabase(Server.mon_hostname, Server.mon_port);
} else if (Server.mon_sid.includes('_HDB')) {
return home.checkHANADatabase(Server.mon_hostname, Server.mon_port);
} else {
return home.checkPort(Server.mon_port, Server.mon_hostname, 1000);
}
});
return Promise.all(operations)
}
const serverlist = [...]
const test = await getList(serverlist)
// test is an array of fulfilled/unfulfilled results of Promise.all
So I would create a function that takes a list of operations(serverList) and returns the promise all like the example above without awaiting for it.
The caller would await it or using another promise all of a series of other calls to your function.
Potentially you can go deeper like Inception :)
// on the caller you can go even deeper
await Promise.all([getList([...]) , getList([...]) , getList([...]) ])
Considering what you added you can return something more customized like:
if (Server.mon_sid.includes('_DB')) {
return home.checkOracleDatabase(Server.mon_hostname, Server.mon_port).then(result => ({result, name: 'Oracle', index:..., date:..., hostname: Server.mon_hostname, port: Server.mon_port}))
In the case above your promise would return a json with the output of your operation as result, plus few additional fields you might want to reuse to organize your data.

For more consistency and for resolving this question i/we(other people which can help you) need all code this all variables definition (because for now i can't find where is the promises variable is defined). Thanks

Wait for async method in for loop in Cypress

Edit: Why this is not a duplicate: because Cypress, just read instead of tagging everything as duplicate.
Edit 2: Also, see answer for better understanding of the differences between usual async for loops problems and this question.
I am writing cypress tests and I want to create a cypress command that populates my database with a list of users. I want the creation loop to wait for each user to be created before it moves on to the next one (because I want that done in a specific order).
For now, my loop looks like this:
Cypress.Commands.add("populateDb", (users) => {
var createdItems = []
for (const user of users) {
cy.createUser(user, 'passe').then(response => {
createdUsers.unshift(response.body.user)
})
}
return createdItems
})
Of course, this loop does not wait for each user to be created before moving onto the next one (I want 'sequential treatment', NOT 'parallel and then wait for all promise to resolve')
I have read the answers about async for-loop here:
JavaScript ES6 promise for loop
Using async/await with a forEach loop
How do I return the response from an asynchronous call?
But I can't seem to find what I want, mainly because cypress wont allow me to declare my function as async as follow :
Cypress.Commands.add("populateDb", async (users) => {
//Some code
})
And If I don't declare it async I am not able to use await.
Isn't there some king of get() method that just synchronously wait for a Promise to resolve?

Using a combination of wrap and each cypress command I was able to achieve a loop that waits for each iteration and can return the full results without needing a global variable.
The only reason I use the wrap command is because the cypress command each requires it to be chained off a previous cypress command. The each command will evaluate each iteration and finally call the then where you can return the complete results array. I am using this to upload multiple files and return the list of keys, but you can modify this for your own need.
Cypress.Commands.add("uploadMultipleFiles", (files) => {
var uploadedS3Keys = []
cy.wrap(files).each((file, index, list) => {
cy.uploadFileToOrdersApi(file)
.then(s3Key => uploadedS3Keys.push(s3Key))
}).then(() => uploadedS3Keys)
})

Turns out there is a reason why cypress is so restrictive about what you can do about waiting for async method to resolve: it always automatically runs all the async commands in sequential order, as they are called, not in parallel, so this will execute in the right order even if createUser is Async :
Cypress.Commands.add("populateDb", (users) => {
for (const user in users) {
cy.createUser(user).then(response => {
console.log(response)
})
}
})
If you want to get the value of what is returned (in my case, I need the user ID to delete them later), you can just store them in a var at file root level and add a cypress command that returns that var.
var createdItems = []
Cypress.Commands.add("getCreatedItems", () => {
return createdItems
})
Cypress.Commands.add("populateDb", (users) => {
for (const user in users) {
cy.createUser(user).then(response => {
createdItems.unshift(response) // I unshift() because I need to delete them in FILO order later
})
}
})
Then in the cypress test file you can just call them in the order you need them to execute :
cy.populateDb(users)
cy.getCreatedItems().then(response => {
//response is my createdItems Array, and cy.getCreatedItems() will run only when cy.populateDb() is resolved
})

You could do it also like:
Cypress.Commands.add("populateDb", (users) => {
for (const user in users) {
cy.createUser(user).then(response => {
createdItems.unshift(response) // I unshift() because I need to delete them in FILO order later
})
}
return createdItems;
})
cy.populateDb(users).then(response => {
// ... runs after populate ...
})
But the other answer is also right. Each cy.xxxxx command is actually added to command queue and ran one after each other. If during queue execution new cy.xxxxx2 commands are called, they will be added to front of the queue.
Here is simple example to explain the execution order of cypress command queue and synchronous code:
const a = 1; // a == 1
cy.cmd1().then(() => {
cy.cmd2().then(() => {
a += 1; // a == 5
});
a += 1; // a == 3
cy.cmd3()
a += 1; // a == 4
});
cy.cmd4().then(() => {
a += 1; // a == 6
});
a += 1; // a == 2
// at this point cypress command queue starts running the queue that is cmd1, cmd4
// 1. cmd1 runs and adds cmd2 and cmd3 to front of command queue also adds +2 to a
// 2. cmd2 runs and adds +1 to a
// 3. cmd3 runs
// 4. cmd4 runs and adds +1 to a
// ... all done ...
So from this example you can see that in your case you loop will be executed serially, because each cy.createUser is added to cypress command queue and executed then sequentially.

Javascript - Call function in parallel and return combined result

This is my first question and I'm trying to learn javascript/nodejs
I have an array x.
var x = [1,2,3,4];
Also I have a function which takes in a param, does some processing and returns a json.
function funcName (param){
//does some external API calls and returns a JSON
return result;
}
Now what I'm looking for is rather than iterating over the array and calling the function again and again, is there a way to call them in parallel and then join the result and return it together ?
Also I'm looking for ways to catch the failed function executions.
for ex: funcName(3) fails for some reason

What you could do is create a file that does your heavy lifting, then run a fork of that file.
In this function we do the following:
loop over each value in the array and create a promise that we will store in an array
Next we create a fork
We then send data to the fork using cp.send()
Wait for a response back and resolve the promise
Using promise.all we can tell when all our child processes have completed
The first parameter will be an array of all the child process results
So our main process will look a little something like this:
const { fork } = require('child_process')
let x = [1,2,3,4]
function process(x) {
let promises = []
for (let i = 0; i < x.length; i++) {
promises.push(new Promise(resolve => {
let cp = fork('my_process.js', [i])
cp.on('message', data => {
cp.kill()
resolve(data)
})
}))
}
Promise.all(promises).then(data => {
console.log(data)
})
}
process(x)
Now in our child we can listen for messages, and do our heavy lifting and return the result back like so (very simple example):
// We got some data lets process it
result = []
switch (process.argv[1]) {
case 1:
result = [1, 1, 1, 1, 1, 1]
break
case 2:
result = [2, 2, 2, 2, 2, 2]
break
}
// Send the result back to the main process
process.send(result)

The comments and other answer are correct. JavaScript has no parallel processing capability whatsoever (forking processes doesn't count).
However, you can make the API calls in a vaguely parallel fashion. Since, as they are asynchronous, the network IO can be interleaved.
Consider the following:
const urls = ['api/items/1', 'api/items/2', etc];
Promise.all(urls.map(fetch))
.then(results => {
processResults(results);
});
While that won't execute JavaScript instructions in parallel, the asynchronous fetch calls will not wait for eachother to complete but will be interleaved and the results will be collected when all have completed.
With error handling:
const urls = ['api/items/1', 'api/items/2', etc];
Promise.all(urls.map(fetch).map(promise => promise.catch(() => undefined))
.then(results => results.filter(result => result !== undefined))
.then(results => {
processResults(results);
});