We Keep Coding

Not able to get results when running HTTPS GET requests on an array of data

I am using the code given below to run multiple https GET request for Wikipedia API.
app.get("/data_results", (req, res) => {
const articlesData = names.map(nameObj => {
let name = nameObj.name;
let articleExtract = "";
let contentURL =
`https://en.wikipedia.org/w/api.php?
action=query&titles=${name}&prop=extracts&format=json&exintro=1&explaintext=false&origin=*`;
// Getting article content
https.get(contentURL, response => {
response.on("data", async data => {
const wikiArticle = JSON.parse(data);
// Mapping the keys of the JSON data of query to its values.
articleExtract = await Object.keys(wikiArticle.query.pages).map(key => wikiArticle.query.pages[key])[0].extract;
nameObj.article = articleExtract.substring(0,350);
})
});
return nameObj;
});
res.send(articlesData);});
This is my names array
[
{ name: 'Indus%20Valley%20Civilisation' },
{ name: 'Ramayana' },
{ name: 'Mahabharata' },
{ name: 'Gautama%20Buddha' }
]
My aim :-
Run the HTTPS GET request for every value in the names array sequentially.
Add the article extract with its name and save in it in an objects array.
You could also suggest me better ways of doing this.

Try this solution. it creates an object like
{
Mahabharata: response
}
The code
const getValues = async () => {
const name_array = [
{name: 'Indus%20Valley%20Civilisation'},
{name: 'Ramayana'},
{name: 'Mahabharata'},
{name: 'Gautama%20Buddha'},
];
const namePromises = name_array.map(value =>
fetch('baseurl' + value.name).then(res => res.json()),
);
const response = await Promise.all(namePromises);
return response.reduce((obj, responseResults, index) => {
obj[name_array[index].name] = responseResults;
}, {});
};

I would suggest you use fetch and promise. all. Map over your array of names and create promises as input. and then return the array of resolved promises.
something like this
const namePromises = [
{ name: 'Indus%20Valley%20Civilisation' },
{ name: 'Ramayana' },
{ name: 'Mahabharata' },
{ name: 'Gautama%20Buddha' }
].map((value) => fetch(baseurl + value.name).then(res=>res.json()));
Promise.all(namePromises).then((response) => {
console.log(response);
});

Problems:
So, you have several things going on here.
http.get() is non-blocking and asynchronous. That means that the rest of your code continues to run while your multiple http.get() operations are running. That means you end up calling res.send(articlesData); before any of the http requests are complete.
.map() is not promise-aware or asynchronous-aware so it will not wait its loop for any asynchronous operation. As such, you're actually running ALL your http requests in parallel (they are all in-flight at the same time). The .map() loop starts them all and then they all finish some time later.
The data event for http.get() is not guaranteed to contain all your data or even a whole piece of data. It may just be a partial chunk of data. So, while your code may seem to get a complete response now, different network or server conditions could change all that and reading the result of your http requests may stop functioning correctly.
You don't show where names comes from in your request handler. If it's a statically declared array of objects declared in a higher-scoped variable, then this code has a problem that you're trying to modify the objects in that array in a request handler and multiple requests to this request handler can be conflicting with one another.
You don't show any error handling for errors in your http requests.
You're using await in a place it doesn't belong. await only does something useful when you are awaiting a promise.
Solutions:
Managing asynchronous operations in Javascript, particularly when you have more than one to coordinate) is a whole lot easier when using promises instead of the plain callback that http.get() uses. And, it's a whole lot easier to use promises when you use an http request interface that already supports promises. My goto library for built-in promise support for http requests in nodejs is got(). There are many good choices shown here.
You can use got() and promises to control the asynchronous flow and error handling like this:
const got = require('got');
app.get("/data_results", (req, res) => {
Promise.all(names.map(nameObj => {
let name = nameObj.name;
// create separate result object
let result = { name };
let contentURL =
`https://en.wikipedia.org/w/api.php?
action=query&titles=${name}&prop=extracts&format=json&exintro=1&explaintext=false&origin=*`;
return got(contentURL).json().then(wikiArticle => {
// Mapping the keys of the JSON data of query to its values.
let articleExtract = Object.keys(wikiArticle.query.pages).map(key => wikiArticle.query.pages[key])[0].extract;
result.article = articleExtract.substring(0, 350);
return result;
});
})).then(articlesData => {
res.send(articlesData);
}).catch(err => {
console.log(err);
res.sendStatus(500);
});
});
This code attempts to fix all six of the above mentioned problems while still running all your http requests in parallel.
If your names array was large, running all these requests in parallel may consume too many resources, either locally or on the target server. Or, it may run into rate limiting on the target server. If that was an issue, you can run sequence the http requests to run them one at a time like this:
const got = require('got');
app.get("/data_results", async (req, res) => {
try {
let results = [];
for (let nameObj of names) {
let name = nameObj.name;
let result = { name };
let contentURL =
`https://en.wikipedia.org/w/api.php?
action=query&titles=${name}&prop=extracts&format=json&exintro=1&explaintext=false&origin=*`;
const wikiArticle = await got(contentURL).json();
let articleExtract = Object.keys(wikiArticle.query.pages).map(key => wikiArticle.query.pages[key])[0].extract;
result.article = articleExtract.substring(0, 350);
results.push(result);
}
res.send(results);
} catch (e) {
console.log(e);
res.sendStatus(500);
}
});
If you really want to stick with https.get(), then you can "promisify" it and use it in place of got():
function myGet(url, options = {}) {
return new Promise((resolve, reject) => {
https.get(url, options, (res) => {
res.setEncoding('utf8');
let rawData = '';
res.on('data', (chunk) => { rawData += chunk; });
res.on('end', () => {
try {
const parsedData = JSON.parse(rawData);
resolve(parsedData);
} catch (e) {
reject(e);
}
});
}).on('error', reject);
});
}
app.get("/data_results", async (req, res) => {
try {
let results = [];
for (let nameObj of names) {
let name = nameObj.name;
let result = { name };
let contentURL =
`https://en.wikipedia.org/w/api.php?
action=query&titles=${name}&prop=extracts&format=json&exintro=1&explaintext=false&origin=*`;
const wikiArticle = await myGet(contentURL);
let articleExtract = Object.keys(wikiArticle.query.pages).map(key => wikiArticle.query.pages[key])[0].extract;
result.article = articleExtract.substring(0, 350);
results.push(result);
}
res.send(results);
} catch (e) {
console.log(e);
res.sendStatus(500);
}
});

Multithreading in node js with bidirectional communication

I have an api which fetches the databases and then on each row of the response i have to do cpu extensive work and then return the response back to the client. So in a nutshell this is what i want to achieve.
exports.updateUser = async(req, res) => {
const users = await fetchUserFromDB(); // 10^4 User
const updatedUser = []
for(const user of users) {
// Do some CPU extensive work here, this takes 100ms
const userDataAfterHeavpOps = await computeUser(user);
updatedUser.push(userDataAfterHeavpOps);
}
return res.send(updatedUser)
}
I have 8 core system, so i am thinking if i can use worker-node/clustering in node this will improve the performance. So this is what i find on the internet
index.js
// run with node --experimental-worker index.js on Node.js 10.x
const { Worker } = require('worker_threads')
function runService(workerData) {
return new Promise((resolve, reject) => {
const worker = new Worker('./service.js', { workerData });
worker.on('message', resolve);
worker.on('error', reject);
worker.on('exit', (code) => {
if (code !== 0)
reject(new Error(`Worker stopped with exit code ${code}`));
})
})
}
async function run() {
const result = await runService(100)
console.log(result);
}
run().catch(err => console.error(err))
worker.js
const { workerData, parentPort } = require('worker_threads')
// You can do any heavy stuff here, in a synchronous way
// without blocking the "main thread"
parentPort.postMessage({ hello: workerData })
Is there any way to send an array of users and process them on different thread to improve the performance and after process them i want to get the computed value so that parent can send it back to client.

How to synchronously serialize api calls in react?

I have a task that requires fetching api data, with the constraint of only one outstanding api request at a time. Must receive a response, or time out, before issuing the next one. Since fetch (or axios) returns a promise, I can’t figure out how to wait for each promise to fulfill before issuing the next fetch.
I'm handed a large array of api url's that must all be resolved in this one-at-a-time manner before continuing.
I’m using create-react-app’s bundled dev server, and Chrome browser.
Curiously, accomplishing this via a node script is easy, because ‘await fetch’ actually waits. Not so in my browser environment, where all the fetch requests blast out at once, returning promises along the way.
Here’s a simple loop that results in the desired behavior as a node script. My question is how to achieve this one-outstanding-request-at-a-time synchronous serialization in the browser environment?
const fetchOne = async (fetchUrl) => {
try {
const response = await fetch(fetchUrl, { // Or axios instead
"headers": {
'accept': 'application/json',
'X-API-Key': 'topSecret'
},
'method': 'GET'
})
const data = await response.json();
if (response.status == 200) {
return (data);
} else {
// error handling
}
} catch(error) {
// different error handling
}
}
const fetchAllData = async (fetchUrlArray) => {
let fetchResponseDataArray = new Array();
let fetchResponseDataObject = new Object(/*object details*/);
for (var j=0; j<fetchUrlArray.length; j++) { // or forEach or map instead
// Node actually synchronously waits between fetchOne calls,
// but react browser environment doesn't wait, instead blasts them all out at once.
// Question is how to achieve the one-outstanding-request-at-a-time synchronous
// serialization in the browser environment?
fetchResponseDataObject = await fetchOne(fetchUrlArray[j]);
fetchResponseDataArray.push(fetchResponseDataObject);
}
return(fetchResponseDataArray);
}

If there's a problem, it's with code you haven't shown (perhaps in one of your components, or maybe in your project configuration).
Here's an runnable example derived from the problem you described, which mocks fetch and an API, showing you how to iterate each network request synchronously (and handle potential errors along the way):
Note, handling potential errors at the boundaries where they might occur is a better practice than only having a top level try/catch: by doing so, you can make finer-grained decisions about what to do in response to each kind of problem. Here, each failed request is stored as [url, error] in a separate array so that you can programmatically make decisions if one or more requests failed. (Maybe you want to retry them in a subsequent step, or maybe you want to show something different in the UI, etc.). Note, there's also Promise.allSettled(), which might be useful to you now or in the future.
<div id="root"></div><script src="https://unpkg.com/react#17.0.2/umd/react.development.js"></script><script src="https://unpkg.com/react-dom#17.0.2/umd/react-dom.development.js"></script><script src="https://unpkg.com/#babel/standalone#7.16.4/babel.min.js"></script>
<script type="text/babel" data-type="module" data-presets="env,react">
const {useEffect, useState} = React;
const successChance = {
fetch: 0.95,
server: 0.95,
};
function mockApi (url, chance = successChance.server) {
// Simulate random internal server issue
const responseArgs = Math.random() < chance
? [JSON.stringify({time: performance.now()}), {status: 200}]
: ['Oops', {status: 500}];
return new Response(...responseArgs);
}
function mockFetch (requestInfo, _, chance = successChance.fetch) {
return new Promise((resolve, reject) => {
// Simulate random network issue
if (Math.random() > chance) {
reject(new Error('Network error'));
return;
}
const url = typeof requestInfo === 'string' ? requestInfo : requestInfo.url;
setTimeout(() => resolve(mockApi(url)), 100);
});
}
// Return an object containing the response if successful (else an Error instance)
async function fetchOne (url) {
try {
const response = await mockFetch(url);
if (!response.ok) throw new Error('Response not OK');
const data = await response.json();
return {data, error: undefined};
}
catch (ex) {
const error = ex instanceof Error ? ex : new Error(String(ex));
return {data: undefined, error};
}
}
async function fetchAll (urls) {
const data = [];
const errors = [];
for (const url of urls) {
const result = await fetchOne(url);
if (result.data) data.push([url, result.data]);
else if (result.error) {
// Handle this however you want
errors.push([url, result.error]);
}
}
return {data, errors};
}
function Example () {
const [data, setData] = useState([]);
const [loading, setLoading] = useState(false);
useEffect(() => {
const fetchData = async () => {
setLoading(true);
try {
const {data, errors} = await fetchAll([
'https://my.url/api/0',
'https://my.url/api/1',
'https://my.url/api/2',
'https://my.url/api/3',
'https://my.url/api/4',
'https://my.url/api/5',
'https://my.url/api/6',
'https://my.url/api/7',
'https://my.url/api/8',
'https://my.url/api/9',
]);
setData(data);
}
catch (ex) {
console.error(ex);
}
setLoading(false);
};
fetchData();
}, []);
return (
<div>
<div>Loading: {loading ? '...' : 'done'}</div>
<ul>
{
data.map(([url, {time}]) => (<li
key={url}
style={{fontFamily: 'monospace'}}
>{url} - {time}</li>))
}
</ul>
</div>
);
}
ReactDOM.render(<Example />, document.getElementById('root'));
</script>

Using Cloud Functions to store data form an external API on Firestore database

I have a Vue.js app running on Firebase and as a database I use Firestore. The app has to import data (clientes) form another app (app2), but app2 only exports by sending an XML code through POST to an address. To receive the POST from app2, my app utilizes Firebase Cloud Functions.
const xml2js = require("xml2js");
const functions = require("firebase-functions");
const cors = require("cors");
const express = require("express");
const app = express();
const admin = require("firebase-admin");
const db = admin.initializeApp().firestore();
function parseXml(xml) {
return new Promise((resolve, reject) => {
xml2js.parseString(xml, { explicitArray: false }, (err, result) => {
if (err) {
reject(err);
} else {
resolve(result);
}
});
});
}
app.use(cors({ origen: true }));
app.post("/", async (request, response) => {
let xml = request.body.XMLRecords.toString();
const clientes = db.collection("clientes");
xml = xml.replace(/&(?!(?:apos|quot|[gl]t|amp);|#)/g, "&").trim();
xml = " " + xml;
console.log("Prep XML:");
console.log(xml);
let parsXml = await parseXml(xml);
console.log("parsXML");
console.log(parsXml);
Array.from(parsXml.Records.Clientes.Cliente).forEach(async cli => {
if (cli !== null) {
delete cli.$;
const docRef = clientes.doc(cli.CliCodigo);
console.log("cli " + cli.CliCodigo);
console.log(cli);
const writeResult = await docRef.set(cli);
console.log("Cliente " + cli.CliCodigo + " salvo");
}
});
response.send("Received.");
});
exports.apiClientes = functions.https.onRequest((request, response) => {
if (!request.path) {
request.url = `/${request.url}`; // prepend '/' to keep query params if any
}
return app(request, response);
});
The app does receive the request and is able to process it, but when I try to comunicate with the Firestore database the function stops sending the console.log()staments and won't save the data to the database.
What am I doing wrong?

This is most probably because within your Array.from(parsXml.Records.Clientes.Cliente).forEach() loop you are executing several asynchronous set() operations but you are returning the response (response.send("Received.");) before those asynchronous operations are done.
By doing response.send("Received."); you are indicating to the instance running your Cloud Function that it can terminate it, but in most cases the asynchronous writes to Firestore are not completed.
You need to correctly handle the Promises returned by the set() method calls, as follows (untested):
//....
const promises = [];
Array.from(parsXml.Records.Clientes.Cliente).forEach(cli => {
if (cli !== null) {
delete cli.$;
const docRef = clientes.doc(cli.CliCodigo);
promises.push(docRef.set(cli));
}
});
await Promise.all(promises);
response.send("Received.");
//....
So, we use Promise.all() to execute in parallel all the set() method calls. Promise.all() method returns a single Promise that fulfills when all of the promises passed to as an iterable (i.e. the promises array) have been fulfilled. Therefore you are sure that all the asynchronous work is done when you send back the response, indicating to the Cloud Function platform that it can safely terminate your Cloud Function.

Don't make functions within a loop no-loop-func Axios Request async await

I have an axios request file that goes and gets many requests which I used an async function around it all so that I could delay the requests per second sent to an api (their rate limit).
My code:
import axios from 'axios';
export const litecoinApi = async (addresses, resolve, reject) => {
let addressesBalance = {};
let addressRequests = [];
addresses.forEach(address => {
addressRequests.push("https://api.blockchair.com/litecoin/dashboards/address/" + address);
});
function delay() {
return new Promise(resolve => {
setTimeout(() => resolve(), 2000);
});
}
let i;
for (i = 0; i < addressRequests.length; i++) {
await axios.get(addressRequests[i])
.then((res) => {
console.log(res.data.data);
const data = res.data.data[addresses[i]];
console.log('data', data.address.balance);
addressesBalance[addresses[i]] = data.address.balance / 100000000;
}).catch((error) => {
console.log(error);
});
await delay();
}
resolve(addressesBalance);
// let i;
// for (i = 0; i < addressRequests.length; i++) {
// await axios.get(addressRequests[i])
// .then((res) => {
// const data = res.data.data;
// console.log(data);
// addressesBalance[data.address.toString()] = data.confirmed_balance.toString();
// }).catch((err) => {
// console.log(err.response);
// });
// await delay();
// }
// resolve(addressesBalance);
};
I am simply accessing the request and going through the object to receive the (address balance).
I am getting the warning in the console:
./src/apis/litecoin.js
Line 20: Don't make functions within a loop no-loop-func
The problem I'm having is that line 20:
.then((res) => {
Is actually not the problem, it is clearly line 22:
const data = res.data.data[addresses[i]];
addresses is simply an array of strings. When I remove accessing the key addresses[i] from the object res.data.data the warning goes away. The object res.data.data in console is:
{LWcXUB6ny88tK49TK1V6KprE5oDcJ1zJhx: {…}}
LWcXUB6ny88tK49TK1V6KprE5oDcJ1zJhx:
address:
{type: null, script_hex: "", balance: 0, balance_usd: 0, received: 0, …}
transactions:
[]
So the litecoin address LWcXUB6ny88tK49TK1V6KprE5oDcJ1zJhx is the key in an object and the value is another object with address and transactions as keys. My final goal is the balance in the value of the address key.
I left a commented for loop below (which was for a slightly different api) because I don't get the warning in that one. The problem is there is simply no way to access the value of the litecoin address key without how you would normally access a key in an object.
I have changed this line:
const data = res.data.data[addresses[i]];
To:
const data = res.data.data;
const address = Object.keys(data);
As soon as I attempt Object.keys I get the error let alone actually accessing the first key:
address[0]
This leaves me at a complete loss because then I literally can't access the object without getting the error that I am using a loop in a function.
The resolve and reject are from a promise that this function gets called in and I use the async function to wait until all the api requests are done before I resolve the promise that this function is in.
That promise then goes out and sets my state correctly.
Anyone have any ideas? I have looked at many other SO however no one seems to be dealing with the problem I am here. I honestly don't see how I am using a function at all in the loop.
Accessing a key in an object certainly isn't a function that I know.
The:
.then((res) => {
...
}
Is simply waiting for the axios.get promise. This get / then cycle has worked in other api requests without this issue. Plus the warning really only happens when attempting to pass a key into an object inside the for loop.
I should let you know everything works correctly with the warning. It just infuriates me that I can't get rid of this warning.
Thank you very much for your time and insight.

You can try something simple and single process at the time:
import axios from 'axios';
export const litecoinApi = async (addresses) => {
const balance = {};
for (const address of addresses) {
const currentAddress = `https://api.blockchair.com/litecoin/dashboards/address/${address}`;
const result = await axios.get(currentAddress)
const data = result.data.data[address];
console.log(data);
balance[address] = data.address.balance / 100000000;
}
return balance;
}
Or you can process all requests in parallel with:
const result = await Promise.all(promises);

On the advice of Teemu ( if you want to re-post this answer i'll give you the answer ).
I tried this:
import axios from 'axios';
export const litecoinApi = async (addresses, resolve, reject) => {
let addressesBalance = {};
let addressRequests = [];
addresses.forEach(address => {
addressRequests.push("https://api.blockchair.com/litecoin/dashboards/address/" + address);
});
function delay() {
return new Promise(resolve => {
setTimeout(() => resolve(), 2000);
});
}
function axiosRequest(addressRequests, addresses) {
axios.get(addressRequests)
.then((res) => {
console.log(res.data.data);
const data = res.data.data[addresses];
console.log('data', data.address.balance);
addressesBalance[addresses] = data.address.balance / 100000000;
}).catch((error) => {
console.log(error);
});
}
let i;
for (i = 0; i < addressRequests.length; i++) {
await axiosRequest(addressRequests[i], addresses[i]);
await delay();
}
resolve(addressesBalance);
};
Which simply moves the axios request into its own function that is then passed the requests and addresses.
The await cannot work within the axiosRequest() function but you can still use it in the for loop solving the problem.
This solved my problem thank you very much Teemu!