We Keep Coding

Await multiple promises to resolve while returning the value of each promise as soon as it is resolved

The problem
While working with a restful API, I had to make multiple requests to retrieve data around a single search. The problem that I seem to be facing is that as the results are returned from a large database, some promises take FOREVER to resolve.
Current solution
Currently I make all the requests in a loop while adding the promises to an array and then using await Promise.all() to wait for them to be resolved but this makes loading times > 30 seconds at times even when the earliest promise resolved within a few seconds.
I am looking for a way that I can 'Lazy Load' the results in. I do have access to the restful server so any changes in either the front-end or back-end would help however I would prefer the changes to be at the front end.
Edit 1
My bad I didn't put any references to the code that I am currently using. For reference here is what I am currently doing
async function retrieve_data() {
let request_urls = [list of api endpoints to fetch]
let promises = []
for (let url of request_urls)
promises.push( fetch(url) )
await promise.all( promises )
// Do something with the returned results
}
The solution I want
async function retrieve_data() {
let request_urls = [list of api endpoints to fetch]
let promises = []
for (let url of request_urls)
promises.push( fetch(url) )
let results = Promise.some_magic_function()
// Or server side
res.write(Promise.some_magic_function()) // If results are received in chunks
// Do something with the returned results
}
Edit 2
So while I did find Promise.any() and Promise.all() to fix half of my problems, I seem unable to comprehend on how to have some function which adds the value of fulfilled promises to an array as soon as they are fulfilled. While these solutions do work for my usecase, I just cant help but think that there must be a better solution.

You can handle the individual results as you're populating the array for Promise.all, which you can use to know when they've all finished.
It's tricky without any code in the question to work with, but here's a simple example:
const promises = [1, 2, 3, 4, 5].map(async (num) => {
const result = await doRequest(num);
console.log(`Immediate processing for "${result}"`);
return result; // If you're going to use the contents of the array from `Promise.all`'s fulfillment
});
const allResults = await Promise.all(promises);
console.log(`All processing done, all results:`);
for (const result of allResults) {
console.log(result);
}
Live Example:
const rndDelay = () => new Promise((resolve) => setTimeout(resolve, Math.round(Math.random() * 1000) + 100));
async function doRequest(num) {
console.log(`Requesting ${num}`);
await rndDelay();
return `Result for ${num}`;
}
async function main() {
const promises = [1, 2, 3, 4, 5].map(async (num) => {
const result = await doRequest(num);
console.log(`Immediate processing for "${result}"`);
return result; // If you're going to use the contents of the array from `Promise.all`'s fulfillment
});
const allResults = await Promise.all(promises);
console.log(`All processing done, all results:`);
for (const result of allResults) {
console.log(result);
}
}
main()
.catch((error) => console.error(error));
.as-console-wrapper {
max-height: 100% !important;
}
Notice that the callback we give map (in this example) is an async function, so it returns a promise and we can use await within it.
If you can't use an async wrapper where you're creating the promises, that isn't a problem, you can fall back to .then:
const promises = [1, 2, 3, 4, 5].map((num) => {
return doRequest(num)
.then((result) => {
console.log(`Immediate processing for "${result}"`);
return result; // If you're going to use the contents of the array from `Promise.all`'s fulfillment
});
});
const allResults = await Promise.all(promises);
console.log(`All processing done, all results:`);
for (const result of allResults) {
console.log(result);
}
const rndDelay = () => new Promise((resolve) => setTimeout(resolve, Math.round(Math.random() * 1000) + 100));
async function doRequest(num) {
console.log(`Requesting ${num}`);
await rndDelay();
return `Result for ${num}`;
}
async function main() {
const promises = [1, 2, 3, 4, 5].map((num) => {
return doRequest(num)
.then((result) => {
console.log(`Immediate processing for "${result}"`);
return result; // If you're going to use the contents of the array from `Promise.all`'s fulfillment
});
});
const allResults = await Promise.all(promises);
console.log(`All processing done, all results:`);
for (const result of allResults) {
console.log(result);
}
}
main()
.catch((error) => console.error(error));
.as-console-wrapper {
max-height: 100% !important;
}

Extract the requests to different async functions and process them the moment they resolve:
// Request that responds in 2 seconds
async function req1() {
let response = await http.get(`${serverRoot}/api/req1`);
// Work on response from req 1. Everything below this line executes after 2'
}
//Request that responds in 30 seconds
async function req2() {
let response = await http.get(`${serverRoot}/api/req2`);
// Work on response from req 1. Everything below this line executes after 30'
}
//Request that responds in 10 seconds
async function req3() {
let response = await http.get(`${serverRoot}/api/req3`);
// Work on response from req 1. Everything below this line executes after 30'
}
Then you can go ahead and add them all in an array and wait for all of them to finish, in order to, let's say, hide a loading indicator:
loading = true;
//Notice that we have to invoke the functions inside the array
await Promise.all([req1(), req2(), req3()]);
//Everything below this line will run after the longest request is finished, 30 seconds
loading = false;

Async/await - passing data to Puppeteer in a MySQL callback

I need to get every client in a table so that I can iterate through them, and use Puppeteer to crawl some data. I need the MySQL query because I gotta pass some params through the querystring.
I'm using Puppeteer, Puppeteer-cluster (due to the hundreds of rows), and MySQL driver.
const mysql = require('mysql');
const { Cluster } = require('puppeteer-cluster');
const sql = "select an.id_clientes, ano, c.nome, c.cpf, c.dt_nasc from clientes_anos an inner join clientes c on an.id_clientes = c.id limit 3";
db.query(sql, async (err, result) => {
//console.log(result);
const cluster = await Cluster.launch({
concurrency: Cluster.CONCURRENCY_CONTEXT,
maxConcurrency: 2,
puppeteerOptions: {
headless: false
},
//monitor: true,
timeout: 90000,
});
for (const elem of result){
const cpf = elem.cpf;
const dt = elem.dt_nasc.toLocaleDateString();
const url = `https://servicos.receita.fazenda.gov.br/Servicos/ConsRest/Atual.app/paginas/index.asp?cpf=${cpf}&dtnasc=${dt}`;
console.log(url);
(async()=>{
cluster.on('taskerror', (err, data) => {
//console.log(`Error crawling ${data}: ${err.message}`);
});
await cluster.task(async ({ page, data: url }) => {
console.log(dt, cpf);
await page.goto(url);
await page.type('#data_nascimento', dt);
await page.type('input[name=CPF]', cpf);
await page.waitForNavigation();
try{
const msg = await page.$eval(
`#rfb-main-container table tbody tr:nth-of-type(1) td:nth-of-type(1)`, divs => divs.innerText.trim()
);
if(msg.includes('Resultado encontrado: Saldo inexistente de imposto a pagar ou a restituir.')){
console.log('situação: '+ msg);
}else{
console.log('0');
}
}catch(exception){
console.log('Error');
}
});
await cluster.queue(url);
await cluster.idle();
await cluster.close();
})();
}
});
The problem I'm having is that when the page loads, await page.type('#data_nascimento', dt); await page.type('input[name=CPF]', cpf); are not returning the right value (the one that comes from the query is right), they are returning the value of the last row. I'm betting it's something with async but I can't figure it out.
Any ideas how I can solve this?

damn boy, i have things to say :)
I think the main cause of your issue is interaction between loops / callbacks / cluster
here is an exemple to clarify my point on loops
for (var i = 0; i<3; i++) {
setTimeout(function() {
console.log(i);
}, 1000);
}
// shows : 3 3 3
// because when console is finally executed "i" has been fully looped
if instead of var you had used const or let, the code would work fine and show 0 1 2.
if you dont use callbacks but only awaits this situation would be much clearer.
async function wait(ms) {
return new Promise(resolve => {
setTimeout(resolve, ms);
});
}
for (var i = 0; i<3; i++) {
await wait(1000);
console.log(i);
}
you can appply this to you mysql call to flatten your code.
function query(sql) {
return new Promise((resolve, reject) => {
db.query(sql, function(err, result) {
if (err) return reject(err);
resolve(result);
})
});
}
//then you call it like this
const sql = "select an.id_clientes, ano, c.nome, c.cpf, c.dt_nasc from clientes_anos an inner join clientes c on an.id_clientes = c.id limit 3";
const result = await query(sql)
in this spirit of flatenning your code, can you remove the async autoexecuting function ? It looks very unnecessary.
"(async()=>{"
...
"})();"
and finally the cluster part : its not written, but from reading the documentation i guess that any data you want to use in the cluster function needs to be pushed in the queue (the XXX).
cluster.queue(XXX)
=>
await cluster.task(async ({ page, data: XXX}) ...)
you should then push not only your url but a whole object if you need many different data.
cluster.queue({cpf, dt, url})
=>
await cluster.task(async ({ page, data: {cpf, dt, url}}) ...)
i never tried using this module, but it makes sens to me.
can you try it ?
PS: Also, not using cluster at all could greatly simplify things.

What are different ways to handle asynchronous calls (apis) inside a for loop or array method?

I have been trying to create an api like this where I tried different things like using array methods like map/filter/reduce where either I get pending promises or result returned before execution of api call.
So my doubt is -->
How do I get total number of drawn matches of all pages ( so I need to add data.total from all pages).
How to better understand this behaviour.
async function getNumDraws(year) {
const goals = [...Array(11).keys()];
let result = 0;
console.log(`before loop ${new Date()}`);
for(let goal of goals){
console.log(`in loop before await ${new Date()}`);
await require('https').get(`https://jsonmock.hackerrank.com/api/football_matches?year=${year}&team1goals=${goal}&team2goals=${goal}`,res=>{
let data="";
res.on('data', (chunk) => {
data += chunk;
});
// The whole res has been received. Print out the result.
res.on('end', () => {
data=JSON.parse(data);
console.log(result,data.total)
result= result + data.total;
});
})
console.log(`in loop after await ${new Date()}`);
}
console.log(`after loop ${new Date()}`);
return result;
}
console.log(getNumDraws(2011));

https.get is a callbacked function so await won't work. You should promisify it first like they did in this other SO question;
const https = require("https"); // only require this once
const getJSONAsync = url => new Promise((resolve, reject) => { // define a function getJSONAsync which returns a Promise that wraps the https.get call, getJSONAsync is awaitable
let req = https.get(url, res => { // make the https.get request
if(res.statusCode < 200 || res.statusCode >= 300) { // if the status code is an error one
return reject(new Error('statusCode=' + res.statusCode)); // reject and skip the rest
}
let body = []; // otherwise accumulate..
res.on('data', chunk => body.push(chunk)); // ..the data
res.on('end', () => { // on end
try {
body = JSON.parse(Buffer.concat(body).toString()); // try to JSON.parse the data
} catch(e) {
reject(e); // reject if an error occurs
}
resolve(body); // resolve the parsed json object otherwise
});
});
req.on("error", error => reject(error)); // reject if the request fails too (if something went wrong before the request is sent for example)
});
async function getNumDraws(year) {
let result = 0;
for(let goal = 0; goal < 11; goal++) {
let data = await getJSONAsync(`https://jsonmock.hackerrank.com/api/football_matches?year=${year}&team1goals=${goal}&team2goals=${goal}`);
result += data.total;
}
return result;
}
Note: getJSONAsync is not specific to getNumDraws, you can use it somewhere else if you need it, and since it returns a Promise you can either await it like getNumDraws does or use it with then/catch blocks like so:
getJSONAsync("url")
.then(data => {
// data is the parsed json returned by the request
})
.catch(error => {
// the error message if something fails
})

Database transactions happen parallely instead of sequentially in loop

I have an endpoint which loops through an array and updates the database as follows.
app.post('/create', Authenticate, async (req, res) => {
const {
products,
} = req.body;
const trxProvider = knex.transactionProvider();
const trx = await trxProvider();
try {
const formattedProduct = await Promise.all(products.map(async (product) => {
// Get Current value
const warehouseProducts = await trx('warehouse_products')
.select('available_qty as qty')
.where('product_code', product.product.code)
.first();
const finalQty = warehouseProducts.qty - product.orderQty;
// Update database
await trx('warehouse_products')
.update({ available_qty: finalQty })
.where('product_code', product.product.code);
}));
await trx('ordered_products')
.insert(formattedProduct);
trx.commit();
console.log('Transaction successful');
return send(res, 201, { success: true });
} catch (err) {
console.log(err);
trx.rollback();
const errors = {};
errors.message = 'something went wrong';
return send(res, 500, errors);
}
});
The issue arises when i try to update the same row of the warehouse_products table within the loop.
In the loop initially the qty value is taken from the warehouse_products table for a particular product then an arithmetic operation is done and the qty value is updated.
Ideally if both iterations access the same row, the second iteration's initial qty value should be what the first iteration updated. However the issue is that the second iteration too starts with the initial value of the first iteration. Its almost as if both iterations are happening parallel to each other instead of occurring sequentially.

Since you are using Promise.all it is supposed to happen in paralle. For sequential processing change this code
await Promise.all(products.map(async (product) => {
// logic here
});
to
for(const product of products) {
// logic here
}

Have a look at the definition for Promise.all()
It is typically used after having started multiple asynchronous tasks to run concurrently and having created promises for their results, so that one can wait for all the tasks being finished.

if you don't want to use an external library like Bluebird or Async
you can go with simple for a loop as following
let delay = (t) => new Promise((resolve) => {
setTimeout(() => {
return resolve(new Date().getMilliseconds())
}, t*1000);
});
let array = [1,1,1,1];
//using simple for loop
async function test() {
let out = [];
for (let i = 0; i < 4; i++) {
const res = await delay(array[i]);
out.push(res);
}
return out;
}
test().then(res => {
console.log(res)
})

Node js how to run axios.get every 2 seconds?

I am kinda of a newbie to node js, Here is what i am trying to do: i am looping through a json file full of links of our website via the map function (around 3000 links), inside the loop i am doing a axios get for each link and getting the response status code(will do other things in the future). But i want to run the axios get only like every 2 seconds or 5 seconds otherwise i am overwhelming the webserver. I am trying to input async await but it's still too fast and server is taking a hit (i am technically DDos-ing my own website). I put a SetTimeout around the axios but that doesn't seem like it worked, since in the console results are printing way too fast. so the question is, how do i make each axios.get request wait every 2 seconds before running in the map loop?.
var axios = require('axios');
const fs = require('fs');
var statusCheck = 0;
var main = [];
let rawdata = fs.readFileSync('C:/Users/jay/Documents/crawl/filtered2.json');
let jsonParsed = JSON.parse(rawdata);
jsonParsed.map(async(line) => {
var encodeLink = encodeURI(line.link);
const response = await axios.get(encodeLink).catch((err) => {
var Status_ErrorsCatchaxios = {
"status Code": err.response.status ? err.response.status : "No status code available",
"Page title:": $('title').text() ? $('title').text() : 'No title avaialble',
"Original Link": encodeLink ? encodeLink : "No Original Link Available",
"errorCode": err
}
main.push(Status_ErrorsCatchaxios)
})
try {
console.log(response.status)
statusCheck = statusCheck + 1;
console.log("Link: ", statusCheck)
} catch (error) {
console.log(error)
}
})

The [].map function doesn't wait for your items to resolve, so your code is currently dispatching all the requests (as you said, around 3000) in parallel.
You can use for...of instead to only run one request at a time. For example:
async function makeRequests (lines) {
for (const line of lines) {
const encodedLink = encodeURI(line.link)
const response = await axios.get(encodedLink)
// ...your response handling code here...
}
}
makeRequests(jsonParsed)
If you want to wait for 2s between each request, you can add this line of code inside your for...of loop:
await new Promise(resolve => setTimeout(resolve, 2000))
Better solution
The solution above works, but I assume your webserver can probably take more than one request at a time, so maybe the ideal scenario would be to limit your code to make only up to N requests in parallel at a given time. This way you don't flood your server but you're able to get your results faster than just doing one request at a time.
The bluebird NPM module allows you to do that with their Promise.map function.
This function receives your list of items as the first argument, a function that executes something and returns a promise for each item as the second argument, and an object with a concurrency key describing how many items you want to allow to be handled in parallel as the third argument.
Here's how it could work:
const bluebird = require('bluebird')
async function makeRequests (lines) {
await bluebird.map(
lines,
async (line) => {
const encodedLink = encodeURI(line.link)
const response = await axios.get(encodedLink)
// ...your response handling code here...
},
{ concurrency: 3 }
)
}
makeRequests(jsonParsed)

Ditch the map, replace with a for ... of, await a promise that takes 2s to resolve, wrap everything inside an async IIFE for the await to be legal.
// dummy data
const fakeJson = new Array(5).fill();
const fakeRequest = () => console.log(`request at ${new Date().toUTCString()}`);
// iteration with 2s in between
(async () => {
for (let line of fakeJson) {
await new Promise(r => setTimeout(r, 2000));
fakeRequest();
}
})()
You can also use more classically use setInterval but HTTP requests are asynchronous, so might as well start with a structure that handles well async and loops.

You are hitting all at once because .map,.forEach,.reduce etc doesn't wait for the Promise to resolve. Use Simple For loop, it will wait for each promise to resolve or reject.
for(let i=0;i<jsonParsed.length;i++) {
var encodeLink = encodeURI(line.link);
const response = await axios.get(encodeLink).catch(...)
try {
....
} catch (error) {
...
}
})
Why it doesn't work?
If we imitate the forEach loop it will be something like,
function forEach(arr, cb){
for(let i=0;i<arr.length;i++){
cb(arr[i], i, cb);
}
}
So you see it doesn't await the cb.

The reason why timeout wont work in a loop is because it will fire all the requests/functions all at once after the timeout delay.
The idea is to put delay in each iteration and only after delay start the next iteration.
you can run a self invoking function which calls itself after the delay. So to run the function every 2 seconds, you can try this:
let jsonParsed = JSON.parse(rawdata);
let len = jsonParsed.length;
(function requestLoop (i) {
setTimeout(function () {
let line = jsonParsed[len-i]
var encodeLink = encodeURI(line.link);
const response = await axios.get(encodeLink).catch((err) => {
var Status_ErrorsCatchaxios = {
"status Code": err.response.status ? err.response.status : "No status code available",
"Page title:": $('title').text() ? $('title').text() : 'No title avaialble',
"Original Link": encodeLink ? encodeLink : "No Original Link Available",
"errorCode": err
}
main.push(Status_ErrorsCatchaxios)
})
try {
console.log(response.status)
statusCheck = statusCheck + 1;
console.log("Link: ", statusCheck)
} catch (error) {
console.log(error)
}
let jsonParsed = JSON.parse(rawdata);
if (--i) requestLoop(i);
}, 2000)
})(len);

You can use
var i=0;
jsonParsed.map(async(line) => {
i++
setTimeout(async() => {
},i*2000)
}

you can use setTimeout function for running codes every 2 second!
setTimeout(async() => {
// await postRequest()
},2000)