We Keep Coding

Not able to get results when running HTTPS GET requests on an array of data

I am using the code given below to run multiple https GET request for Wikipedia API.
app.get("/data_results", (req, res) => {
const articlesData = names.map(nameObj => {
let name = nameObj.name;
let articleExtract = "";
let contentURL =
`https://en.wikipedia.org/w/api.php?
action=query&titles=${name}&prop=extracts&format=json&exintro=1&explaintext=false&origin=*`;
// Getting article content
https.get(contentURL, response => {
response.on("data", async data => {
const wikiArticle = JSON.parse(data);
// Mapping the keys of the JSON data of query to its values.
articleExtract = await Object.keys(wikiArticle.query.pages).map(key => wikiArticle.query.pages[key])[0].extract;
nameObj.article = articleExtract.substring(0,350);
})
});
return nameObj;
});
res.send(articlesData);});
This is my names array
[
{ name: 'Indus%20Valley%20Civilisation' },
{ name: 'Ramayana' },
{ name: 'Mahabharata' },
{ name: 'Gautama%20Buddha' }
]
My aim :-
Run the HTTPS GET request for every value in the names array sequentially.
Add the article extract with its name and save in it in an objects array.
You could also suggest me better ways of doing this.

Try this solution. it creates an object like
{
Mahabharata: response
}
The code
const getValues = async () => {
const name_array = [
{name: 'Indus%20Valley%20Civilisation'},
{name: 'Ramayana'},
{name: 'Mahabharata'},
{name: 'Gautama%20Buddha'},
];
const namePromises = name_array.map(value =>
fetch('baseurl' + value.name).then(res => res.json()),
);
const response = await Promise.all(namePromises);
return response.reduce((obj, responseResults, index) => {
obj[name_array[index].name] = responseResults;
}, {});
};

I would suggest you use fetch and promise. all. Map over your array of names and create promises as input. and then return the array of resolved promises.
something like this
const namePromises = [
{ name: 'Indus%20Valley%20Civilisation' },
{ name: 'Ramayana' },
{ name: 'Mahabharata' },
{ name: 'Gautama%20Buddha' }
].map((value) => fetch(baseurl + value.name).then(res=>res.json()));
Promise.all(namePromises).then((response) => {
console.log(response);
});

Problems:
So, you have several things going on here.
http.get() is non-blocking and asynchronous. That means that the rest of your code continues to run while your multiple http.get() operations are running. That means you end up calling res.send(articlesData); before any of the http requests are complete.
.map() is not promise-aware or asynchronous-aware so it will not wait its loop for any asynchronous operation. As such, you're actually running ALL your http requests in parallel (they are all in-flight at the same time). The .map() loop starts them all and then they all finish some time later.
The data event for http.get() is not guaranteed to contain all your data or even a whole piece of data. It may just be a partial chunk of data. So, while your code may seem to get a complete response now, different network or server conditions could change all that and reading the result of your http requests may stop functioning correctly.
You don't show where names comes from in your request handler. If it's a statically declared array of objects declared in a higher-scoped variable, then this code has a problem that you're trying to modify the objects in that array in a request handler and multiple requests to this request handler can be conflicting with one another.
You don't show any error handling for errors in your http requests.
You're using await in a place it doesn't belong. await only does something useful when you are awaiting a promise.
Solutions:
Managing asynchronous operations in Javascript, particularly when you have more than one to coordinate) is a whole lot easier when using promises instead of the plain callback that http.get() uses. And, it's a whole lot easier to use promises when you use an http request interface that already supports promises. My goto library for built-in promise support for http requests in nodejs is got(). There are many good choices shown here.
You can use got() and promises to control the asynchronous flow and error handling like this:
const got = require('got');
app.get("/data_results", (req, res) => {
Promise.all(names.map(nameObj => {
let name = nameObj.name;
// create separate result object
let result = { name };
let contentURL =
`https://en.wikipedia.org/w/api.php?
action=query&titles=${name}&prop=extracts&format=json&exintro=1&explaintext=false&origin=*`;
return got(contentURL).json().then(wikiArticle => {
// Mapping the keys of the JSON data of query to its values.
let articleExtract = Object.keys(wikiArticle.query.pages).map(key => wikiArticle.query.pages[key])[0].extract;
result.article = articleExtract.substring(0, 350);
return result;
});
})).then(articlesData => {
res.send(articlesData);
}).catch(err => {
console.log(err);
res.sendStatus(500);
});
});
This code attempts to fix all six of the above mentioned problems while still running all your http requests in parallel.
If your names array was large, running all these requests in parallel may consume too many resources, either locally or on the target server. Or, it may run into rate limiting on the target server. If that was an issue, you can run sequence the http requests to run them one at a time like this:
const got = require('got');
app.get("/data_results", async (req, res) => {
try {
let results = [];
for (let nameObj of names) {
let name = nameObj.name;
let result = { name };
let contentURL =
`https://en.wikipedia.org/w/api.php?
action=query&titles=${name}&prop=extracts&format=json&exintro=1&explaintext=false&origin=*`;
const wikiArticle = await got(contentURL).json();
let articleExtract = Object.keys(wikiArticle.query.pages).map(key => wikiArticle.query.pages[key])[0].extract;
result.article = articleExtract.substring(0, 350);
results.push(result);
}
res.send(results);
} catch (e) {
console.log(e);
res.sendStatus(500);
}
});
If you really want to stick with https.get(), then you can "promisify" it and use it in place of got():
function myGet(url, options = {}) {
return new Promise((resolve, reject) => {
https.get(url, options, (res) => {
res.setEncoding('utf8');
let rawData = '';
res.on('data', (chunk) => { rawData += chunk; });
res.on('end', () => {
try {
const parsedData = JSON.parse(rawData);
resolve(parsedData);
} catch (e) {
reject(e);
}
});
}).on('error', reject);
});
}
app.get("/data_results", async (req, res) => {
try {
let results = [];
for (let nameObj of names) {
let name = nameObj.name;
let result = { name };
let contentURL =
`https://en.wikipedia.org/w/api.php?
action=query&titles=${name}&prop=extracts&format=json&exintro=1&explaintext=false&origin=*`;
const wikiArticle = await myGet(contentURL);
let articleExtract = Object.keys(wikiArticle.query.pages).map(key => wikiArticle.query.pages[key])[0].extract;
result.article = articleExtract.substring(0, 350);
results.push(result);
}
res.send(results);
} catch (e) {
console.log(e);
res.sendStatus(500);
}
});

Using async when two requests have to be made and response from the first is needed in the other request

I have a async function like this:
exports.myFunction = async (req, res, next) => {
if (some condition) {
next()
}
try {
const results = await axios.get(`https://a-domain.com/url/path`);
const info = results.data;
const docRef = await addDoc(collection(db, "name_of_collec"), info);
console.log("Document written with ID: ", docRef.id);
} catch (e) {
console.error("Error adding document: ", e);
}
next();
};
Here, response from the first request is required in the second request. But according to my knowledge since I am making asynchronous requests, the two requests should be made at the same time.
What I cannot understand:
So, how it is possible that there is no error occurring?
How is it that, sometimes the no document is added to Firestore? Like it is skipped somehow...
What modification can be done so that await addDoc(collection(db, "name_of_collec") executes compulsorily?
Context info:
I am using Express.js and Node.js to make a backend
I am using Axios to make the first request and the second one is adding a document to Firestore.
Firestore is a NoSQL database inside Google's Firebase.

But according to my knowledge since I am making asynchronous requests,
the two requests should be made at the same time.
asynchronous doesn't mean that the requests are made at the same time. It means the code won't wait for the responses from these requests. Requests are made in the manner you add them in the code and responses will be asynchronous i.e the response will be received at some point in future time.
And there is no guarantee in what manner they will be received.
However, when you sue async/await, the code execution waits at await keyword for the response based on its success or failure it calls the next flow. It is nothing but syntactic sugar on promises. Under the hood, this is how your code will be executed:
exports.myFunction = async (req, res, next) => {
if (some condition) {
next()l
}
axios.get(`https://a-domain.com/url/path`).then(result => result).then(data => {
const info = data;
addDoc(collection(db, "name_of_collec"), info).then(response => {
console.log("Document written with ID: ", docRef.id);
}).catch(e => {
console.error(e);
})
}).catch(e => {
console.error(e);
})
next();
};
So, how it is possible that there is no error occurring?
It is not necessary that the error occurs when you request. If that is the case catch handler should be activated.
How is it that, sometimes the no document is added to Firestore? Like
it is skipped somehow...
The reason might be info is an empty object and firestore automatically removes the empty document.
There is no issue, but there is not data info that is empty sometimes that skips the addition(it does request firestore, but firestore removes the empty docs)
exports.myFunction = async (req, res, next) => {
if (some condition) {
next()
}
try {
const results = await axios.get(`https://a-domain.com/url/path`);
const info = results.data;
const is_Info_Data = Array.isArray(info) ? info?.length >= 1: Object.keys(info) !== 0
if (is_Info_Data) {
const docRef = await addDoc(collection(db, "name_of_collec"), info);
console.log("Document written with ID: ", docRef.id);
} else {
console.log(`Info is empty ${info} no addition to DB`);
}
next();
} catch (e) {
console.error("Error adding document: ", e);
next(e);
}
};

using async/await for sql data insert

function Recursive_scan_and_Insert (path_dir) { //scanning path_dir recursively and insert filepath into temporary list
Recursive_Scan(path_dir, (err, files) => { //it's from npm recursive_readdir
if(err) {
console.log(err);
res.status(500).send('server error');
}
files.forEach(elements => {
let params = [elements]; //
DB("GET", "INSERT INTO filelist_t VALUES (null, ?, NOW(), 0, 0)", params).then(function(res) {
console.log('data input');
});
});
});
};
function Add_to_DB () { //moving temporal list to main list without duplicate
DB("GET", "INSERT INTO filelist (id, path, addeddate, isdeleted, ismodified) SELECT NULL, filelist_t.path, filelist_t.addeddate, filelist_t.isdeleted, filelist_t.ismodified FROM filelist_t LEFT JOIN filelist ON filelist.path = filelist_t.path WHERE filelist.id IS NULL; DELETE FROM filelist_t; ").then(function(res) {
console.log('data moving');
});
};
app.get('/db', (req, res) => { //PROBLEM PART
async function async_Two_Functions () {
var object_path = '/want/to/scan/path';
await Recursive_scan_and_Insert(object_path).then( () => {
return Add_to_DB()
})
}
async_Two_Functions();
res.send(res);
});
app.get('/dbp', (req, res) => { //show main list to my web
DB("GET", "SELECT * FROM filelist").then(function(res2) {
res.send(res2.row);
});
});
here's the thing.
there are 4 stage in my dream algorithm.
recursively scan all the path.
insert each data into temporary table.
moving temporal data on main table, without duplicate
present main table
it's very important to things get order. but I don't understand about async await exactly...

Well, here's a cleaned up version of the code with lots of changes.
const {promisify} = require('util');
const Recursive_ScanP = promisify(Recursive_Scan);
function Recursive_scan_and_Insert(path_dir) { //scanning path_dir recursively and insert filepath into temporary list
return Recursive_ScanP(path_dir).then(files => {
return Promise.all(files.map(elements => {
let params = [elements];
return DB("GET", "INSERT INTO filelist_t VALUES (null, ?, NOW(), 0, 0)", params).then(function(res) {
console.log('data input');
// what should the return value be here?
});
}));
});
};
function Add_to_DB () { //moving temporal list to main list without duplicate
return DB("GET", "INSERT INTO filelist (id, path, addeddate, isdeleted, ismodified) SELECT NULL, filelist_t.path, filelist_t.addeddate, filelist_t.isdeleted, filelist_t.ismodified FROM filelist_t LEFT JOIN filelist ON filelist.path = filelist_t.path WHERE filelist.id IS NULL; DELETE FROM filelist_t; ").then(function(res) {
console.log('data moving');
return res;
});
};
app.get('/db', async (req, res) => {
try {
let object_path = '/want/to/scan/path';
await Recursive_scan_and_Insert(object_path);
await Add_to_DB();
res.send(somethingHere); // you fill in what response you want to send here
} catch(e) {
console.log(e);
res.status(500).send("Server Error");
}
});
app.get('/dbp', (req, res) => { //show main list to my web
DB("GET", "SELECT * FROM filelist").then(function(res2) {
res.send(res2.row);
}).catch(err => {
console.log(err);
res.status(500).send("Server Error");
});
});
Changes:
Return a promise from every function that has an asynchronous operation in it
Return whatever you want the resolved value to be for the promise from every .then() handler
Promisify anything that uses a regular callback so you can do all control flow with promises
Use Promise.all() to know when multiple promises are done and to collect results in order from doing a set of asynchronous operation in parallel
Use async/await as desired, but particularly when you want to sequence multiple asynchronous operatoins
Use try/catch around any await to catch rejected promise that aren't being returned to a higher level where they will be caught
Use .catch() with any .then() that isn't being returned to a higher level where it will be caught
Open Questions:
Don't know what response you want to send from app.get('/db', ...). You will have to fill that in.
Are you expecting any resolved value from Recursive_scan_and_Insert()?
Does Add_to_DB() really accept no input? It just reorganizes things already in the database?

Database Querying function not running asynchronously

So I created a function in my node server which takes in a query string, runs it on my db, and returns the results. I then wanted to use my function asynchronously using async await throughout my routes instead of having nested query within, nested query, within nested query etc.
So here is the code:
const runQuery = queryString => {
console.log("...runQuery")
db.query(queryString, (error, results, fields) => {
if (error) {
console.log("runQuery: FAILURE");
return error;
}
else {
console.log("runQuery: SUCCESS");
return(results);
}
})
}
register.post("/", async (req, res) => {
console.log(req.body);
const results = await runQuery("select * from test1");
res.send(results);
})
The database should have 3 entries, but unfortunately, it returns nothing. Meaning results is an empty variable when it is sent, meaning JS never properly waits for it to capture the db results. How can I use my function asynchronously, and how is this even feasible?

It seems your function "runQuery" does not return a promise, in fact, it's not returning anything. You are using "return" in the callback of the db.query function, not the function "runQuery" itself.
Since runQuery is performing an asynchronous operation, the result ought to be resolved via a promise (which is what the "await" in your request handler is looking for).
I'm not exactly sure but it seems you are using MySql, and I could not find anything on the npm page of the mysql package regarding the query being promisified, so we'll promisify it ourselves:
const runQuery = (queryString) => new Promise((resolve,reject) => {
console.log("...runQuery")
db.query(queryString, (error, results, fields) => {
if (error) {
console.error("runQuery: FAILURE");
reject(error);
} else {
console.log("runQuery: SUCCESS");
resolve(results);
}
})
})
register.post("/", async (req, res) => {
console.log(req.body);
try{
const results = await runQuery("select * from test1");
res.send(results);
}catch(e){
console.error(`ERROR THROWN BY runQuery:`,e);
res.status(500).send({
message: e.message || "INTERNAL SERVER ERROR"
})
}
})
Note that if an error occurs, our promisified function will reject the error and it will NOT be stored in the "results" variable in our request handler. Instead, an error will be thrown which needs to be handled. Therefore, it is always a good practice to put any async/await calls inside a try/catch.

How to run request sequentially in a loop?

This is a pseudo code of what I am trying to achieve. First I need to get a list of URLs from the request body then pass those URLs to request function (using request module) which will get the data from each url and then save those data to MongoDB. After all the requests are finished including saving data to the server only then it should send a response.
app.post('/', (req, resp) => {
const { urls } = req.body;
urls.forEach((url, i) => {
request(url, function (err, resp, body) {
if (err) {
console.log('Error: ', err)
} else {
// function to save data to MongoDB server
saveUrlData(body);
console.log(`Data saved for URL number - ${i+1}`)
}
})
});
// Should be called after all data saved from for loop
resp.send('All data saved')
})
I have tried this code and of course the resp.send() function will run without caring if the request has completed. Using this code I get a result on the console like this:
Data saved for URL number - 3
Data saved for URL number - 1
Data saved for URL number - 5
Data saved for URL number - 2
Data saved for URL number - 4
I could write them in nested form but the variable urlscan have any number of urls and that's why it needs to be in the loop at least from my understanding. I want the requests to run sequentially i.e. it should resolve 1st url and then second and so on and when all urls are done only then it should respond. Please help!

app.post('/', async (req, resp) => {
const {
urls
} = req.body;
for (const url of urls) {
try {
const result = await doRequest(url)
console.log(result)
} catch (error) {
// do error processing here
console.log('Error: ', err)
}
}
})
function doRequest(url) {
return new Promise((resolve, reject) => {
request(url, function(err, resp, body) {
err ? reject(err) ? resolve(body)
})
})
}
using async await

You should look at JavaScript Promises
Otherwise, you can do a recursive request like so:
app.post('/', (req, resp) => {
const { urls } = req.body;
sendRequest(urls, 0);
})
function sendRequest(urlArr, i){
request(urlArr[i], function (err, resp, body) {
if (err) {
console.log('Error: ', err)
}
else {
saveUrlData(body);
console.log(`Data saved for URL number - ${i+1}`)
}
i++;
if(i == urlArr.length) resp.send('All data saved') //finish
else sendRequest(urlArr, i); //send another request
})
}
All I had to do is create a separate function I can call over and over again, passing the url array and a base index 0 as arguments. Each success callback increments the index variable which I pass in the same function again. Rinse and repeat until my index hits the length of the url array, I'll stop the recursive loop from there.

You want to wait till all api response you get and stored in db, so you should do async-await and promisify all the response.
You can use Request-Promise module instead of request. So you will get promise on every requested api call instead of callback.
And use promise.all for pushing up all request(module) call inside array.
Using async-await you code execution will wait till all api call get response and stored in db.
const rp = require('request-promise');
app.post('/', async (req, res) => {
try{
const { urls } = req.body;
// completed all will have all the api resonse.
const completedAll = await sendRequest(urls);
// now we have all api response that needs to be saved
// completedAll is array
const saved = await saveAllData(completedAll);
// Should be called after all data saved from for loop
res.status(200).send('All data saved')
}
catch(err) {
res.status(500).send({msg: Internal_server_error})
}
})
function sendRequest(urlArr, i){
const apiCalls = [];
for(let i=0;i < urlArr.length; i++){
apiCalls.push(rp(urlArr[i]));
}
// promise.all will give all api response in order as we pushed api call
return Promise.all(apiCalls);
}
You can refer these links:
https://www.npmjs.com/package/request-promise
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all

Looking at the intention(a crawler) you can use Promise.all because the urls are not dependant upon each other.
app.post('/', (req, resp) => {
const { urls } = req.body;
const promises = urls.map((url, i) => {
return new Promise((resolve, rej)=>{
request(url, function (err, resp, body) {
if (err) {
rej(err);
} else {
resolve(body);
}
})
})
.then((body)=>{
//this should definitely be a promise as you are saving data to mongo
return saveUrlData(body);
})
});
// Should be called after all data saved from for loop
Promise.all(promises).then(()=>resp.send('All data saved'));
})
Note: Need to do error handling as well.

there are multiple ways to solve this.
you can use async/await
Promises
you can also use the async library
app.post('/', (req, res, next) => {
const { urls } = req.body;
async.each(urls, get_n_save, err => {
if (err) return next(err);
res.send('All data saved');
});
function get_n_save (url, callback) {
request(url, (err, resp, body) => {
if (err) {
return callback(err);
}
saveUrlData(body);
callback();
});
}
});