We Keep Coding

Not able to get results when running HTTPS GET requests on an array of data

I am using the code given below to run multiple https GET request for Wikipedia API.
app.get("/data_results", (req, res) => {
const articlesData = names.map(nameObj => {
let name = nameObj.name;
let articleExtract = "";
let contentURL =
`https://en.wikipedia.org/w/api.php?
action=query&titles=${name}&prop=extracts&format=json&exintro=1&explaintext=false&origin=*`;
// Getting article content
https.get(contentURL, response => {
response.on("data", async data => {
const wikiArticle = JSON.parse(data);
// Mapping the keys of the JSON data of query to its values.
articleExtract = await Object.keys(wikiArticle.query.pages).map(key => wikiArticle.query.pages[key])[0].extract;
nameObj.article = articleExtract.substring(0,350);
})
});
return nameObj;
});
res.send(articlesData);});
This is my names array
[
{ name: 'Indus%20Valley%20Civilisation' },
{ name: 'Ramayana' },
{ name: 'Mahabharata' },
{ name: 'Gautama%20Buddha' }
]
My aim :-
Run the HTTPS GET request for every value in the names array sequentially.
Add the article extract with its name and save in it in an objects array.
You could also suggest me better ways of doing this.

Try this solution. it creates an object like
{
Mahabharata: response
}
The code
const getValues = async () => {
const name_array = [
{name: 'Indus%20Valley%20Civilisation'},
{name: 'Ramayana'},
{name: 'Mahabharata'},
{name: 'Gautama%20Buddha'},
];
const namePromises = name_array.map(value =>
fetch('baseurl' + value.name).then(res => res.json()),
);
const response = await Promise.all(namePromises);
return response.reduce((obj, responseResults, index) => {
obj[name_array[index].name] = responseResults;
}, {});
};

I would suggest you use fetch and promise. all. Map over your array of names and create promises as input. and then return the array of resolved promises.
something like this
const namePromises = [
{ name: 'Indus%20Valley%20Civilisation' },
{ name: 'Ramayana' },
{ name: 'Mahabharata' },
{ name: 'Gautama%20Buddha' }
].map((value) => fetch(baseurl + value.name).then(res=>res.json()));
Promise.all(namePromises).then((response) => {
console.log(response);
});

Problems:
So, you have several things going on here.
http.get() is non-blocking and asynchronous. That means that the rest of your code continues to run while your multiple http.get() operations are running. That means you end up calling res.send(articlesData); before any of the http requests are complete.
.map() is not promise-aware or asynchronous-aware so it will not wait its loop for any asynchronous operation. As such, you're actually running ALL your http requests in parallel (they are all in-flight at the same time). The .map() loop starts them all and then they all finish some time later.
The data event for http.get() is not guaranteed to contain all your data or even a whole piece of data. It may just be a partial chunk of data. So, while your code may seem to get a complete response now, different network or server conditions could change all that and reading the result of your http requests may stop functioning correctly.
You don't show where names comes from in your request handler. If it's a statically declared array of objects declared in a higher-scoped variable, then this code has a problem that you're trying to modify the objects in that array in a request handler and multiple requests to this request handler can be conflicting with one another.
You don't show any error handling for errors in your http requests.
You're using await in a place it doesn't belong. await only does something useful when you are awaiting a promise.
Solutions:
Managing asynchronous operations in Javascript, particularly when you have more than one to coordinate) is a whole lot easier when using promises instead of the plain callback that http.get() uses. And, it's a whole lot easier to use promises when you use an http request interface that already supports promises. My goto library for built-in promise support for http requests in nodejs is got(). There are many good choices shown here.
You can use got() and promises to control the asynchronous flow and error handling like this:
const got = require('got');
app.get("/data_results", (req, res) => {
Promise.all(names.map(nameObj => {
let name = nameObj.name;
// create separate result object
let result = { name };
let contentURL =
`https://en.wikipedia.org/w/api.php?
action=query&titles=${name}&prop=extracts&format=json&exintro=1&explaintext=false&origin=*`;
return got(contentURL).json().then(wikiArticle => {
// Mapping the keys of the JSON data of query to its values.
let articleExtract = Object.keys(wikiArticle.query.pages).map(key => wikiArticle.query.pages[key])[0].extract;
result.article = articleExtract.substring(0, 350);
return result;
});
})).then(articlesData => {
res.send(articlesData);
}).catch(err => {
console.log(err);
res.sendStatus(500);
});
});
This code attempts to fix all six of the above mentioned problems while still running all your http requests in parallel.
If your names array was large, running all these requests in parallel may consume too many resources, either locally or on the target server. Or, it may run into rate limiting on the target server. If that was an issue, you can run sequence the http requests to run them one at a time like this:
const got = require('got');
app.get("/data_results", async (req, res) => {
try {
let results = [];
for (let nameObj of names) {
let name = nameObj.name;
let result = { name };
let contentURL =
`https://en.wikipedia.org/w/api.php?
action=query&titles=${name}&prop=extracts&format=json&exintro=1&explaintext=false&origin=*`;
const wikiArticle = await got(contentURL).json();
let articleExtract = Object.keys(wikiArticle.query.pages).map(key => wikiArticle.query.pages[key])[0].extract;
result.article = articleExtract.substring(0, 350);
results.push(result);
}
res.send(results);
} catch (e) {
console.log(e);
res.sendStatus(500);
}
});
If you really want to stick with https.get(), then you can "promisify" it and use it in place of got():
function myGet(url, options = {}) {
return new Promise((resolve, reject) => {
https.get(url, options, (res) => {
res.setEncoding('utf8');
let rawData = '';
res.on('data', (chunk) => { rawData += chunk; });
res.on('end', () => {
try {
const parsedData = JSON.parse(rawData);
resolve(parsedData);
} catch (e) {
reject(e);
}
});
}).on('error', reject);
});
}
app.get("/data_results", async (req, res) => {
try {
let results = [];
for (let nameObj of names) {
let name = nameObj.name;
let result = { name };
let contentURL =
`https://en.wikipedia.org/w/api.php?
action=query&titles=${name}&prop=extracts&format=json&exintro=1&explaintext=false&origin=*`;
const wikiArticle = await myGet(contentURL);
let articleExtract = Object.keys(wikiArticle.query.pages).map(key => wikiArticle.query.pages[key])[0].extract;
result.article = articleExtract.substring(0, 350);
results.push(result);
}
res.send(results);
} catch (e) {
console.log(e);
res.sendStatus(500);
}
});

How to make API calls for each element in array

I'm using Next.js / React.js. I'm using this API to get a specific country.
There's an array on this response called borders, for example.
borders: [
"CAN",
"MEX",
],
There's an end point to get the data based on a border, for example.
https://restcountries.eu/rest/v2/alpha/can
How would I get the data both borders, i.e. each element in the borders array? It's two API calls that I've tried to make in a loop, but I get undefined.
export async function getServerSideProps(context) {
const { name } = context.params;
const res = await fetch(`https://restcountries.eu/rest/v2/name/${name}?fullText=true`)
const countryRes = await res.json();
const country = countryRes[0];
// Get the borders
const borders = country.borders;
// I'm making an API call to each element in array
const borderCountr = borders.forEach(border => {
fetch(`https://restcountries.eu/rest/v2/alpha/${border}`);
});
console.log(borderCountr); // undefinded
if (!country) {
return {
notFound: true,
}
}
return {
props: { country }
}
}

A good approach would be using Promise.all, to be sure that each fetch is correctly executed. Also, you need to make those calls async. something like:
const borderCountr = await Promise.all(
borders.map(async (border) => {
const response = await fetch(`https://restcountries.eu/rest/v2/alpha/${border}`);
return await response.json();
})
);
console.log(borderCountr[0], borderCountr[1]);

// I'm making an API call to each element in array
const borderCountr = borders.forEach(border => {
fetch(`https://restcountries.eu/rest/v2/alpha/${border}`);
});
This is not awaited (you do not await for ANY of the fetch results) that mens, the code executes here, and without waiting for the fetches to finish - executed the next lines.
As forEach returns undefined, that is your variable content.

Variable is not storing Promise result in Node [duplicate]

This question already has answers here:
Why is my variable unaltered after I modify it inside of a function? - Asynchronous code reference
(7 answers)
How do I access previous promise results in a .then() chain?
(17 answers)
Closed 2 years ago.
I'm developing a script that connects with an API, then with the JSON reponse do some operations and then reformat the JSON to send it to another API.
But now I'm stuck in the first step as I can't deal with the first part as my Promises is not working as expected. How can I store the API's response into a variable? For development puropose I stored one API response into a JSON file. This is my code:
declare var require: any;
let url = './data/big buy/big-bui-product-list.json';
const fs = require('fs');
let counter = 0;
const getProductList = () => {
return new Promise((resolve, reject) => {
fs.readFile(url, 'utf8', (err, data) => {
if(err){
return reject (err);
}
else {
return resolve(JSON.parse(data));
}
})
})
}
const getProductStock = () => {
return new Promise((resolve, reject) => {
fs.readFile('./data/big buy/big-bui-product-stock.json', 'utf8', (err, data) => {
if(err) {
return reject(err);
}
else {
return resolve(JSON.parse(data));
}
})
})
}
try {
let products;
console.log('Products:');
Promise.all([getProductList()])
.then(function(result) {
products = result[0];
});
console.log('Stocks:');
const productStock = Promise.all([getProductStock()]);
console.log(products);
}
catch(e) {
console.log((`Ha ocurrido un error: ${e.message}`));
}
finally {
}
In this code, what I do is getting a list of products and then get the stocks of all the products, later I will add a new function that will filter by stock and get me just a list of products where stock is bigger than X units. Now when I launch it from the terminal I dont' get the response stored into products variable but if I add .then((data) => console.log(data)) into the Promise I see on screen the JSON but as I dont' have it stored it in any variable I don't see how can I work with the objects I'm retrieving.

Promises are asynchronous. They are quite literally promises that a value will be yielded in the future. When you do getProductList().then(x => products = x), you're saying that Javascript should fetch the product list in the background, and once it's finished, it should set the products variable to x. The key words there are "in the background", since the code afterwards keeps running while the product list is being fetched. The products variable is only guaranteed to be set after the .then portion is run. So, you can move things into the .then:
try {
let products;
getProductList().then(list => {
products = list;
return getProductStock(); // leverage promise chaining
}).then(stocks => {
console.log("Products:", products);
console.log("Stocks:", stocks);
});
}
catch(e) {
console.log((`Ha ocurrido un error: ${e.message}`));
}
finally {
}

You seem to be missing some fundamental knowledge about promises. I suggest reading through the MDN Using Promises guide to familiarize yourself a bit with them.
A structure like below never works.
let variable;
promise.then(data => variable = data);
console.log(variable);
This is because it is executed in the following manner.
Create the variable variable.
Add a promise handler to the promise.
Log variable.
Promise gets resolved.
Execute the promise handler.
Set variable variable to data.
If you are using Node.js 10 or higher you can use the promise API of file system to simplify your code quite a bit.
const fs = require('fs/promises');
const readJSON = (path) => fs.readFile(path, "utf8").then((json) => JSON.parse(json));
const getProductList = () => readJSON("./data/big buy/big-bui-product-list.json");
const getProductStock = () => readJSON("./data/big buy/big-bui-product-stock.json");
Promise.all([
getProductList(),
getProductStock(),
]).then(([products, stocks]) => {
console.log({ products, stocks });
}).catch((error) => {
console.log("Ha ocurrido un error:", error.message);
}).finally(() => {
// ...
});

Putting s3 object data into an array - node

I'm trying to fetch data from an S3 object and put it into an array. I plan to map through this array and display the data on a React front end in grid/list whatever. I'm struggling with nested functions though, so I'd appreciate some help.
const dataFromS3 = async (bucket, file) => {
let lines = [];
const options = {
Bucket: bucket,
Key: file
};
s3.getObject(options, (err, data) => {
if (err) {
console.log(err);
} else {
let objectData = data.Body.toString('utf-8');
lines.push(objectData);
console.log(lines);
return lines;
}
});
};
Formatting is a bit weird but this is my function to get data from s3. I want to take the output of this function in the form of an array and pass it to my '/' route which I'm testing:
app.get('/', async (req, res, next) => {
try {
let apolloKey = await dataFromS3(s3Bucket, apolloKeywords);
res.send(apolloKey);
} catch (err) {
console.log('Error: ', err);
}
});
It seems that the return value of lines in the s3.getObject function needs to be returned within the first function so that I can access it in app.get but I can't seem to do it after some attempts. The value in lines turns into an empty array if I return it at the end of datafromS3() and I can't find a way to return it. I've tried using promises also using a method found here - How to get response from S3 getObject in Node.js? but I get a TypeError: Converting Circular Structure to JSON...
Thank you

You need to make your dataFromS3 func like htis. You were not returning anything from that. AWS also provided promise based function.
const dataFromS3 = async (bucket, file) => {
const lines = [];
const options = {
"Bucket": bucket,
"Key": file
};
const data = await s3.getObject(options).promise();
const objectData = data.Body.toString("utf-8");
lines.push(objectData); // You might need to conversion here using JSON.parse(objectData);
console.log(lines);
return lines;
};

Multiple Async call resolutions using Promises

In Node.js, I have a Promise.all(array) resolution with a resulting value that I need to combine with the results of another asynchronous function call. I am having problems getting the results of this second function, since it resolves later than the promise.all. I could add it to the Promise.all, but it would ruin my algorithm. Is there a way to get these values outside of their resolutions so I can modify them statically? Can I create a container that waits for their results?
To be more specific, I am reading from a Firebase realtime database that has been polling API data. I need to run an algorithm on this data and store it in a MongoDB archive. But the archive opens aynchronously and I can't get it to open before my results resolve (which need to be written).
Example:
module.exports = {
news: async function getNews() {
try {
const response = await axios.get('https://cryptopanic.com/api/posts/?auth_token=518dacbc2f54788fcbd9e182521851725a09b4fa&public=true');
//console.log(response.data.results);
var news = [];
response.data.results.forEach((results) => {
news.push(results.title);
news.push(results.published_at);
news.push(results.url);
});
console.log(news);
return news;
} catch (error) {
console.error(error);
}
},
coins: async function resolution() {
await Promise.all(firebasePromise).then((values) => {
//code
return value
}
}
I have tried the first solution, and it works for the first entry, but I may be writing my async function wrong on my second export, because it returns undefined.

You can return a Promise from getNews
module.exports = {
news: function getNews() {
return axios.get('https://cryptopanic.com/api/posts/?auth_token=518dacbc2f54788fcbd9e182521851725a09b4fa&public=true')
.then(res => {
// Do your stuff
return res;
})
}
}
and then
let promiseSlowest = news();
let promiseOther1 = willResolveSoon1();
let promiseOther2 = willResolveSoon2();
let promiseOther3 = willResolveSoon3();
Promise.all([ promiseOther1, promiseOther2, promiseOther3 ]).then([data1,
data2, data3] => {
promiseSlowest.then(lastData => {
// data1, data2, data3, lastData all will be defined here
})
})
The benefit here is all promises will start and run concurrently so your total waiting time will be equal to the time taken by the promiseSlowest.
Check the link for more:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function#Examples