We Keep Coding

Not able to push data in an empty array, I'm getting that data from an external api, using https module in Node js

var https = require("https");
const arr = [];
for (let i = 1; i < 26; i++) {
https.get(
`https://jsonmock.hackerrank.com/api/countries?page=${i}`,
(res) => {
res.on("data", (data) => {
JSON.parse(data).data.map((info, i) => {
let { name } = info;
arr.push(name);
console.log(name);
});
});
}
);
}
console.log(arr);
when I'm just logging JSON.parse(data) I'm getting the required data on my console
but When I'm trying to push it into an array it's not happening instead it logs an empty array onto the console
really need to know the reason as I'm stuck with this for 3 days now

Your issue is that the callback to https.get - i.e. (res) => is called asynchronously - therefore, console.log(arr); is executed before the 25 https.get requests are even made - you'll see that arr.push does actually work if you console.log(arr); where you console.log(name); - so your assumption that you are not able to push is incorrect
You are pushing to the array, you just never console.log the array when it has data in it
I guess one way I can suggest doing this with rather old version of Node.js you are using (14.16.0) is as follows
var https = require("https");
function fn(callback) {
const result = [];
let done = 0;
for (let i = 1; i < 26; i++) {
https.get(`https://jsonmock.hackerrank.com/api/countries?page=${i}`, (res) => {
res.on("data", (data) => {
JSON.parse(data).data.map((info) => {
let { name } = info;
result.push(name);
console.log(name);
});
// keep track of how many requests have "completed"
done = done + 1;
// when all are done, call the callback
if (done === 25) {
callback(result);
}
});
});
}
}
fn(arr => { // here is where arr is populated, nowhere else
console.log(arr);
});
Note, arr will only be accessible inside the callback - you will NOT be able to access arr at the top-level like you want - the only possible way you could is to use a version of Node.js that supports top-level await - and convert the code to use Promise (which is a trivial task)
Not sure it's important, but there's no guarantee that the names in arr will be in the correct order ... i.e. the results from iteration 3 may be pushed after iteration 4, for example, since that's the nature of network requests, there is no guarantee when they will finish
As an aside. If you were to use the latest (18.1 at the time of writing) version of node.js - the above can be written like
const promises = Array.from({length:25}, async (_, i) => {
const res = await fetch(`https://jsonmock.hackerrank.com/api/countries?page=${i+1}`);
const data = await res.json();
return data.map(({name}) => name);
});
const arr = (await Promise.all(promises)).flat(2);
console.log(arr);
Things of note are
native fetch - which makes networks requests simple compared to regular node.js methods
top-level await - so, you can use arr at the top-level of the code
6 lines of code vs over 20

Promises: keep track of matching input/output identifiers

Say I need to fire several requests via an API, in order to store some data in a database. Every entry has a unique identifier on the client side. However, upon insertion on the remote side, it will get a new unique identifier. This cannot be changed, i.e. I cannot force the remote side to use the same identifiers. This identifier (along with some other data) is sent back to the client when the Promise resolves. What is the best practice to keep track of it all.
Illustration:
let inputValues = ["id1", "id2", "id3"];
for (val of inputValues) {
api.insert(val).then( (result) => {
console.log("new id:", result.id);
});
}
In the end, I imagine having maybe an associative array like
[ "id1": "new ID for id1 from remote server",
"id2": "new ID for id2 from remote server",
"id3": "new ID for id3 from remote server" ]
I am pretty confident that I could write up something that would do the job, but that would probably be awful code full of anti-patterns. So I prefer to ask first: what would be the recommended way to do it?

It looks like you're doing your updates in parallel (rather than in series), so you could use Promise.allSettled, which accepts an iterable (like an array) of promises, waits for all of them to settle (get fulfilled or rejected), and then returns an array in the same order as the iterable you provided to it. You can then loop through and, for the successful updates, apply the new ID.
Something like this (in an async function):
const inputValues = ["id1", "id2", "id3"];
const results = await Promise.allSettled(
inputValues.map(value => api.insert(value))
);
// Here, `results` and `inputValues` will be parallel arrays
for (let i = 0; i < results.length; ++i) {
const result = results[i];
if (result.status === "fulfilled") {
const newId = result.value.id;
// Successful update, `newId` is the new ID for `inputValues[i]`
}
}
Here's an example, with the promises intentionally being settled out of order to demonstrate that the result array is in the same order as the input iterable (since you weren't sure that was the case):
const api = {
async insert(value) {
const delay = value === "id2" ? 1000 : 200;
await new Promise(resolve => setTimeout(resolve, delay));
console.log(`Fulfilling ${JSON.stringify(value)}`);
return {
id: `New ID for ${value}`
};
}
};
(async () => {
const inputValues = ["id1", "id2", "id3"];
const results = await Promise.allSettled(
inputValues.map(value => api.insert(value))
);
// Here, `results` and `inputValues` will be parallel arrays
for (let i = 0; i < results.length; ++i) {
const result = results[i];
if (result.status === "fulfilled") {
const newId = result.value.id;
const input = inputValues[i];
console.log(`input value = ${JSON.stringify(input)}, newId = ${JSON.stringify(newId)}`);
}
}
})();
In that you can see that even though the operation for "id2" took longer than the ones for "id1" and "id3", it's still in the second position in the result.
If for some reason you can't use an async function:
const inputValues = ["id1", "id2", "id3"];
Promise.allSettled(
inputValues.map(value => api.insert(value))
)
.then(results => {
// Here, `results` and `inputValues` will be parallel arrays
for (let i = 0; i < results.length; ++i) {
const result = results[i];
if (result.status === "fulfilled") {
const newId = result.value.id;
// Successful update, `newId` is the new ID for `inputValues[i]`
}
}
})
.catch(error => {
// ...handle/report error...
});

How to make multiple nested http requests with promises and async/await while web scraping

Below I have a Node.js function that makes a series of requests to different urls, then for each url I use the Cheerio web scraping library to loop through elements on the dom and create a sub array. At the end of each request (after the sub array is full) I'd like to push the contents of that array to a larger array, which is outside of the request scope.
The approach I'm trying doesn't seem to be working. It looks like I don't have access to 'allPlayers' from inside the .then block.
function readPlayers(teamUrls){
const allPlayers = [];
teamUrls.forEach((teamUrl, i) => {
const options = {
gzip: true,
uri: teamUrl,
Connection: 'keep-alive',
transform: function (body) {
return cheerio.load(body);
}
};
request(options)
.then(($) => {
const team = [];
$('tbody').children('tr').each(function(j, element){
const playerName = $(element).children('td').eq(1).children('span').eq(1).find('a').text().trim();
const player = { 'playerName': playerName };
team.push(player);
});
allPlayers.push(team);
}).catch(err => console.log("error: " + err)) );
});
}
So I'm wondering the best way to re-write this code to make the requests work and populate the outer array (allPlayers) with the results.
I've looked into trying to push the entire request directly into the outer array, to no avail.
In this example I'm using request-promise to make the request.
I've looked into using Promise.map, which I think is suited for this situation. Then I would return the entire request (I think), but I don't exactly understand what I'm doing in that case.. or if it will work.
Could anyone explain the scoping in this case, why I can't do it like I'm trying.
Many thanks

You have to remember when you are using asynchronous function you cannot go back to synchronous code execution.
This is one of the methods you can do it. It will fetch all the players parallely:
async function readPlayers(teamUrls) {
const playerPromises = teamUrls.map((teamUrl, i) => {
const options = {
gzip: true,
uri: teamUrl,
Connection: 'keep-alive',
transform: function(body) {
return cheerio.load(body);
}
};
return request(options)
});
const players = await Promise.all(playerPromises);
return players.reduce((allPlayers, $) =>{
const team = [];
$('tbody').children('tr').each(function(j, element) {
const playerName = $(element).children('td').eq(1).children('span').eq(1).find('a').text().trim();
const player = { playerName: playerName };
team.push(player);
});
allPlayers.push(team);
return allPlayers;
},[])
}
And you can use it using await readPlayers(array) or readPlayers(array).then(allteamplayers=>{...})
Note: In the current code it will be a 2D array, [[{p1:p1}..], [{p2:p2}..]] etc

If you use a forEach, every callback will run asynchronously and you won't be able to await them. You could swap it to a for loop, collect your promises in an array and then await the completion of all of them:
async function readPlayers(teamUrls) {
const allPlayers = [];
const allPromises = [];
for (var i = 0; i < teamUrls.length; i++) {
var teamUrl = teamUrls[i];
const options = {
gzip: true,
uri: teamUrl,
Connection: "keep-alive",
transform: function(body) {
return cheerio.load(body);
}
};
allPromises.push(
request(options)
.then($ => {
const team = [];
$("tbody")
.children("tr")
.each(function(j, element) {
const playerName = $(element)
.children("td")
.eq(1)
.children("span")
.eq(1)
.find("a")
.text()
.trim();
const player = { playerName: playerName };
team.push(player);
});
allPlayers.push(team);
})
.catch(err => console.log("error: " + err))
);
// wait untill all the promises resolve
await Promise.all(allPromises);
console.log(allPlayers);
return allPlayers;
}
}
Then you can get all the players by awaiting your function:
var allPlayers = await readPlayers(teamUrls);

Run HTTP requests in chunks

I want to run 1 thundered http requests in configurable chunks, and set configurable timeout between chunk requests. The request is based on the data provided with some.csv file.
It doesn't work because I am getting a TypeError, but when I remove () after f, it doesn't work either.
I would be very grateful for a little help. Probably the biggest problem is that I don't really understand how exactly promises work, but I tried multiple solutions and I wasn't able to achieve what I want.
The timeout feature will probably give me even more headache so I would appreciate any tips for this too.
Can you please help me to understand why it doesn't work?
Here is the snippet:
const rp = require('request-promise');
const fs = require('fs');
const { chunk } = require('lodash');
const BATCH_SIZE = 2;
const QUERY_PARAMS = ['clientId', 'time', 'changeTime', 'newValue'];
async function update(id, time, query) {
const options = {
method: 'POST',
uri: `https://requesturl/${id}?query=${query}`,
body: {
"prop": {
"time": time
}
},
headers: {
"Content-Type": "application/json"
},
json: true
}
return async () => { return await rp(options) };
}
async function batchRequestRunner(data) {
const promises = [];
for (row of data) {
row = row.split(',');
promises.push(update(row[0], row[1], QUERY_PARAMS.join(',')));
}
const batches = chunk(promises, BATCH_SIZE);
for (let batch of batches) {
try {
Promise.all(
batch.map(async f => { return await f();})
).then((resp) => console.log(resp));
} catch (e) {
console.log(e);
}
}
}
async function main() {
const input = fs.readFileSync('./input.test.csv').toString().split("\n");
const requestData = input.slice(1);
await batchRequestRunner(requestData);
}
main();
Clarification for the first comment:
I have a csv file which looks like below:
clientId,startTime
123,13:40:00
321,13:50:00
the file size is ~100k rows
the file contains information how to update time for a particular clientId in the database. I don't have an access to the database but I have access to an API which allows to update entries in the database.
I cannot make 100k calls at once, because: my network is limited (I work remotely because of coronavirus), it comsumpts a lot of memory, and API can also be limited and can crash if I will make all the requests at once.
What I want to achieve:
Load csv into memory, convert it to an Array
Handle api requests in chunks, for example take first two rows from the array, make API call based on the first two rows, wait 1000ms, take another two rows, and continue processing until the end of array (csv file)

Well, it seems like this is a somewhat classic case of where you want to process an array of values with some asynchronous operation and to avoid consuming too many resources or overwhelming the target server, you want to have no more than N requests in-flight at the same time. This is a common problem for which there are pre-built solutions for. My goto solution is a small piece of code called mapConcurrent(). It's analagous to array.map(), but it assumes a promise-returning asynchronous callback and you pass it the max number of items that should ever be in-flight at the same time. It then returns to you a promise that resolves to an array of results.
Here's mapConcurrent():
// takes an array of items and a function that returns a promise
// returns a promise that resolves to an array of results
function mapConcurrent(items, maxConcurrent, fn) {
let index = 0;
let inFlightCntr = 0;
let doneCntr = 0;
let results = new Array(items.length);
let stop = false;
return new Promise(function(resolve, reject) {
function runNext() {
let i = index;
++inFlightCntr;
fn(items[index], index++).then(function(val) {
++doneCntr;
--inFlightCntr;
results[i] = val;
run();
}, function(err) {
// set flag so we don't launch any more requests
stop = true;
reject(err);
});
}
function run() {
// launch as many as we're allowed to
while (!stop && inflightCntr < maxConcurrent && index < items.length) {
runNext();
}
// if all are done, then resolve parent promise with results
if (doneCntr === items.length) {
resolve(results);
}
}
run();
});
}
Your code can then be structured to use it like this:
function update(id, time, query) {
const options = {
method: 'POST',
uri: `https://requesturl/${id}?query=${query}`,
body: {
"prop": {
"time": time
}
},
headers: {
"Content-Type": "application/json"
},
json: true
}
return rp(options);
}
function processRow(row) {
let rowData = row.split(",");
return update(rowData[0], rowData[1], rowData[2]);
}
function main() {
const input = fs.readFileSync('./input.test.csv').toString().split("\n");
const requestData = input.slice(1);
// process this entire array with up to 5 requests "in-flight" at the same time
mapConcurrent(requestData, 5, processRow).then(results => {
console.log(results);
}).catch(err => {
console.log(err);
});
}
You can obviously adjust the number of concurrent requests to whatever number you want. I set it to 5 here in this example.

How to wait all Promises to resolve in ExpressJS?

Background:
I have a simple REST API in ExpressJS that allows phasing multiple pages together. The page numbers are dynamic.
Issue:
Because of performance constraints, I would like to implement async promises when fetching multiple web pages, wait for all of them finished download, phrase them with my desired format, and return back to the output.
After I researched all the stuff about promises and async online (since I'm still new to this async topic), Most of them told me to use Promise.all, but I just can't get it to work somehow.
Current output when navigating to GET /xxx
{
username: xxx
parsedHTML: []
}
Targeted output:
{
username: xxx
parsedHTML: [
"BUNCH OF ANALYSED HTML",
"BUNCH OF ANALYSED HTML",
"BUNCH OF ANALYSED HTML",
...
]
}
Code
const express = require("express");
const http = require("http");
const fetch = require('node-fetch');
const app = express();
app.get("/:username", (req, res)=>{
const username = req.params.username;
let page = 3; //Will be obtained dynamically. But for now, I'll put a constant here
res.json({
username: username,
parsedHTML: getParsedHTML(username, page),
});
console.log("Page sent")
})
function getParsedHTML(username, page) {
let promises = [];
let analyses = [];
for (var i = 1; i < (page + 1); i++) {
promises.push(fetch(`https://example.com/profile/${username}/?page=${i}`)
.then((c) => c.text()));
// console.log(`Added promise`)
}
Promise.all(promises).then(()=>{
for (let i = 0; i < promises.length; i++) {
let promise = promises[i];
analyses.push(analyse(promise));
}
})
return analyses;
}
function analyse(html){
// Some synchronous analyse stuff here
// Right now it do nothing
return html;
}
app.listen(3000, () => console.log('API listening on port ' + 3000 + '!'))
Any helps would be appreciated. Thanks a lot.

You're calling Promise.all on the promises correctly, but the getParsedHTML function isn't waiting for that Promise.all call to resolve before returning. So, your res.json is running immediately, synchronously, and the analyses that are returned is an empty array.
Return the Promise.all call instead, and make sure to analyze the responses (from the Promise.all call) rather than the Promises:
return Promise.all(promises).then((responses)=>{
for (let i = 0; i < responses.length; i++) {
let response = responses[i];
analyses.push(analyse(response));
}
}).then(() => analyses);
But you can significantly clean up your code by mapping the resulting array of responses:
function getParsedHTML(username, page) {
let promises = [];
for (var i = 1; i < (page + 1); i++) {
promises.push(fetch(`https://example.com/profile/${username}/?page=${i}`)
.then((c) => c.text()));
}
return Promise.all(promises)
.then((responses) => responses.map(analyse));
}
Also make sure for to wait for getParsedHTML to wait for the returned Promises to resolve before sending res.json:
app.get("/:username", (req, res)=>{
const username = req.params.username;
let page = 3; //Will be obtained dynamically. But for now, I'll put a constant here
getParsedHTML(username, page)
.then((parsedHTML) => {
res.json({
username,
parsedHTML
});
console.log("Page sent")
});
})