We Keep Coding

Not able to get results when running HTTPS GET requests on an array of data

I am using the code given below to run multiple https GET request for Wikipedia API.
app.get("/data_results", (req, res) => {
const articlesData = names.map(nameObj => {
let name = nameObj.name;
let articleExtract = "";
let contentURL =
`https://en.wikipedia.org/w/api.php?
action=query&titles=${name}&prop=extracts&format=json&exintro=1&explaintext=false&origin=*`;
// Getting article content
https.get(contentURL, response => {
response.on("data", async data => {
const wikiArticle = JSON.parse(data);
// Mapping the keys of the JSON data of query to its values.
articleExtract = await Object.keys(wikiArticle.query.pages).map(key => wikiArticle.query.pages[key])[0].extract;
nameObj.article = articleExtract.substring(0,350);
})
});
return nameObj;
});
res.send(articlesData);});
This is my names array
[
{ name: 'Indus%20Valley%20Civilisation' },
{ name: 'Ramayana' },
{ name: 'Mahabharata' },
{ name: 'Gautama%20Buddha' }
]
My aim :-
Run the HTTPS GET request for every value in the names array sequentially.
Add the article extract with its name and save in it in an objects array.
You could also suggest me better ways of doing this.

Try this solution. it creates an object like
{
Mahabharata: response
}
The code
const getValues = async () => {
const name_array = [
{name: 'Indus%20Valley%20Civilisation'},
{name: 'Ramayana'},
{name: 'Mahabharata'},
{name: 'Gautama%20Buddha'},
];
const namePromises = name_array.map(value =>
fetch('baseurl' + value.name).then(res => res.json()),
);
const response = await Promise.all(namePromises);
return response.reduce((obj, responseResults, index) => {
obj[name_array[index].name] = responseResults;
}, {});
};

I would suggest you use fetch and promise. all. Map over your array of names and create promises as input. and then return the array of resolved promises.
something like this
const namePromises = [
{ name: 'Indus%20Valley%20Civilisation' },
{ name: 'Ramayana' },
{ name: 'Mahabharata' },
{ name: 'Gautama%20Buddha' }
].map((value) => fetch(baseurl + value.name).then(res=>res.json()));
Promise.all(namePromises).then((response) => {
console.log(response);
});

Problems:
So, you have several things going on here.
http.get() is non-blocking and asynchronous. That means that the rest of your code continues to run while your multiple http.get() operations are running. That means you end up calling res.send(articlesData); before any of the http requests are complete.
.map() is not promise-aware or asynchronous-aware so it will not wait its loop for any asynchronous operation. As such, you're actually running ALL your http requests in parallel (they are all in-flight at the same time). The .map() loop starts them all and then they all finish some time later.
The data event for http.get() is not guaranteed to contain all your data or even a whole piece of data. It may just be a partial chunk of data. So, while your code may seem to get a complete response now, different network or server conditions could change all that and reading the result of your http requests may stop functioning correctly.
You don't show where names comes from in your request handler. If it's a statically declared array of objects declared in a higher-scoped variable, then this code has a problem that you're trying to modify the objects in that array in a request handler and multiple requests to this request handler can be conflicting with one another.
You don't show any error handling for errors in your http requests.
You're using await in a place it doesn't belong. await only does something useful when you are awaiting a promise.
Solutions:
Managing asynchronous operations in Javascript, particularly when you have more than one to coordinate) is a whole lot easier when using promises instead of the plain callback that http.get() uses. And, it's a whole lot easier to use promises when you use an http request interface that already supports promises. My goto library for built-in promise support for http requests in nodejs is got(). There are many good choices shown here.
You can use got() and promises to control the asynchronous flow and error handling like this:
const got = require('got');
app.get("/data_results", (req, res) => {
Promise.all(names.map(nameObj => {
let name = nameObj.name;
// create separate result object
let result = { name };
let contentURL =
`https://en.wikipedia.org/w/api.php?
action=query&titles=${name}&prop=extracts&format=json&exintro=1&explaintext=false&origin=*`;
return got(contentURL).json().then(wikiArticle => {
// Mapping the keys of the JSON data of query to its values.
let articleExtract = Object.keys(wikiArticle.query.pages).map(key => wikiArticle.query.pages[key])[0].extract;
result.article = articleExtract.substring(0, 350);
return result;
});
})).then(articlesData => {
res.send(articlesData);
}).catch(err => {
console.log(err);
res.sendStatus(500);
});
});
This code attempts to fix all six of the above mentioned problems while still running all your http requests in parallel.
If your names array was large, running all these requests in parallel may consume too many resources, either locally or on the target server. Or, it may run into rate limiting on the target server. If that was an issue, you can run sequence the http requests to run them one at a time like this:
const got = require('got');
app.get("/data_results", async (req, res) => {
try {
let results = [];
for (let nameObj of names) {
let name = nameObj.name;
let result = { name };
let contentURL =
`https://en.wikipedia.org/w/api.php?
action=query&titles=${name}&prop=extracts&format=json&exintro=1&explaintext=false&origin=*`;
const wikiArticle = await got(contentURL).json();
let articleExtract = Object.keys(wikiArticle.query.pages).map(key => wikiArticle.query.pages[key])[0].extract;
result.article = articleExtract.substring(0, 350);
results.push(result);
}
res.send(results);
} catch (e) {
console.log(e);
res.sendStatus(500);
}
});
If you really want to stick with https.get(), then you can "promisify" it and use it in place of got():
function myGet(url, options = {}) {
return new Promise((resolve, reject) => {
https.get(url, options, (res) => {
res.setEncoding('utf8');
let rawData = '';
res.on('data', (chunk) => { rawData += chunk; });
res.on('end', () => {
try {
const parsedData = JSON.parse(rawData);
resolve(parsedData);
} catch (e) {
reject(e);
}
});
}).on('error', reject);
});
}
app.get("/data_results", async (req, res) => {
try {
let results = [];
for (let nameObj of names) {
let name = nameObj.name;
let result = { name };
let contentURL =
`https://en.wikipedia.org/w/api.php?
action=query&titles=${name}&prop=extracts&format=json&exintro=1&explaintext=false&origin=*`;
const wikiArticle = await myGet(contentURL);
let articleExtract = Object.keys(wikiArticle.query.pages).map(key => wikiArticle.query.pages[key])[0].extract;
result.article = articleExtract.substring(0, 350);
results.push(result);
}
res.send(results);
} catch (e) {
console.log(e);
res.sendStatus(500);
}
});

Using the response of a Axios/NodeJS API call function as a parameter to make another API call

General concept: I am accessing two API endpoints. One responds with general information, and the other with specific information.
I have called the first endpoint and with the response, I return an object in the format that the second endpoint requires as a parameter.
Problem: I keep getting "Data after transformation must be a string, an ArrayBuffer, a Buffer, or a Stream"
I want to learn to use functions across controllers, so as not having to REPEAT myself.
Code:
This returns the object that I need to use in the second API call:
const axios = require("axios");
const getSpecials = async (req, res) => {
try {
const response = await axios.get(`${apiURL}/categories`);
let specials = [];
let skus = [];
response.data.forEach((element, index) => {
if (response.data[index].is_special == true) {
specials.push({
name: response.data[index].name,
product_skus: response.data[index].product_skus,
_id: response.data[index]._id,
});
skus.push(response.data[index].product_skus);
}
});
return { product_skus: skus.flat() };
} catch (error) {
console.log(error.message);
}
};
module.exports = {getSpecials}
This returns the product details if I post the return from the first API call as a parameter:
const axios = require("axios");
const { getSpecials } = require("./categoriesControllers");
const specialProducts = async (req, res) => {
try {
const response = await axios.post(`${apiURL}/products`, getSpecials);
res.status(200).json(response.data);
} catch (error) {
console.log(error.message);
}
};
Any suggestions on how best to do this would be great. I want learn to let the backend to do all the heavy lifting and I want to serve the data as cleanly as possible to the front end.

If I understand correctly, you just need to change
const response = await axios.post(`${apiURL}/products`, getSpecials);
to this
const response = await axios.post(`${apiURL}/products`, await getSpecials());
because right now you're passing a function declaration to second argument to axios post function, not a return value from getSpecials

How to synchronously serialize api calls in react?

I have a task that requires fetching api data, with the constraint of only one outstanding api request at a time. Must receive a response, or time out, before issuing the next one. Since fetch (or axios) returns a promise, I can’t figure out how to wait for each promise to fulfill before issuing the next fetch.
I'm handed a large array of api url's that must all be resolved in this one-at-a-time manner before continuing.
I’m using create-react-app’s bundled dev server, and Chrome browser.
Curiously, accomplishing this via a node script is easy, because ‘await fetch’ actually waits. Not so in my browser environment, where all the fetch requests blast out at once, returning promises along the way.
Here’s a simple loop that results in the desired behavior as a node script. My question is how to achieve this one-outstanding-request-at-a-time synchronous serialization in the browser environment?
const fetchOne = async (fetchUrl) => {
try {
const response = await fetch(fetchUrl, { // Or axios instead
"headers": {
'accept': 'application/json',
'X-API-Key': 'topSecret'
},
'method': 'GET'
})
const data = await response.json();
if (response.status == 200) {
return (data);
} else {
// error handling
}
} catch(error) {
// different error handling
}
}
const fetchAllData = async (fetchUrlArray) => {
let fetchResponseDataArray = new Array();
let fetchResponseDataObject = new Object(/*object details*/);
for (var j=0; j<fetchUrlArray.length; j++) { // or forEach or map instead
// Node actually synchronously waits between fetchOne calls,
// but react browser environment doesn't wait, instead blasts them all out at once.
// Question is how to achieve the one-outstanding-request-at-a-time synchronous
// serialization in the browser environment?
fetchResponseDataObject = await fetchOne(fetchUrlArray[j]);
fetchResponseDataArray.push(fetchResponseDataObject);
}
return(fetchResponseDataArray);
}

If there's a problem, it's with code you haven't shown (perhaps in one of your components, or maybe in your project configuration).
Here's an runnable example derived from the problem you described, which mocks fetch and an API, showing you how to iterate each network request synchronously (and handle potential errors along the way):
Note, handling potential errors at the boundaries where they might occur is a better practice than only having a top level try/catch: by doing so, you can make finer-grained decisions about what to do in response to each kind of problem. Here, each failed request is stored as [url, error] in a separate array so that you can programmatically make decisions if one or more requests failed. (Maybe you want to retry them in a subsequent step, or maybe you want to show something different in the UI, etc.). Note, there's also Promise.allSettled(), which might be useful to you now or in the future.
<div id="root"></div><script src="https://unpkg.com/react#17.0.2/umd/react.development.js"></script><script src="https://unpkg.com/react-dom#17.0.2/umd/react-dom.development.js"></script><script src="https://unpkg.com/#babel/standalone#7.16.4/babel.min.js"></script>
<script type="text/babel" data-type="module" data-presets="env,react">
const {useEffect, useState} = React;
const successChance = {
fetch: 0.95,
server: 0.95,
};
function mockApi (url, chance = successChance.server) {
// Simulate random internal server issue
const responseArgs = Math.random() < chance
? [JSON.stringify({time: performance.now()}), {status: 200}]
: ['Oops', {status: 500}];
return new Response(...responseArgs);
}
function mockFetch (requestInfo, _, chance = successChance.fetch) {
return new Promise((resolve, reject) => {
// Simulate random network issue
if (Math.random() > chance) {
reject(new Error('Network error'));
return;
}
const url = typeof requestInfo === 'string' ? requestInfo : requestInfo.url;
setTimeout(() => resolve(mockApi(url)), 100);
});
}
// Return an object containing the response if successful (else an Error instance)
async function fetchOne (url) {
try {
const response = await mockFetch(url);
if (!response.ok) throw new Error('Response not OK');
const data = await response.json();
return {data, error: undefined};
}
catch (ex) {
const error = ex instanceof Error ? ex : new Error(String(ex));
return {data: undefined, error};
}
}
async function fetchAll (urls) {
const data = [];
const errors = [];
for (const url of urls) {
const result = await fetchOne(url);
if (result.data) data.push([url, result.data]);
else if (result.error) {
// Handle this however you want
errors.push([url, result.error]);
}
}
return {data, errors};
}
function Example () {
const [data, setData] = useState([]);
const [loading, setLoading] = useState(false);
useEffect(() => {
const fetchData = async () => {
setLoading(true);
try {
const {data, errors} = await fetchAll([
'https://my.url/api/0',
'https://my.url/api/1',
'https://my.url/api/2',
'https://my.url/api/3',
'https://my.url/api/4',
'https://my.url/api/5',
'https://my.url/api/6',
'https://my.url/api/7',
'https://my.url/api/8',
'https://my.url/api/9',
]);
setData(data);
}
catch (ex) {
console.error(ex);
}
setLoading(false);
};
fetchData();
}, []);
return (
<div>
<div>Loading: {loading ? '...' : 'done'}</div>
<ul>
{
data.map(([url, {time}]) => (<li
key={url}
style={{fontFamily: 'monospace'}}
>{url} - {time}</li>))
}
</ul>
</div>
);
}
ReactDOM.render(<Example />, document.getElementById('root'));
</script>

Firestore took too long to fetch data therefore I couldn't display it

I have a function which fetch a data from Firestore :
getLastTime(collectionName: string) {
const docRef = this.afs.firestore.collection(collectionName).doc(this.User).collection('lastTime').doc('lastTime');
docRef.get().then(doc => {
if (doc.exists) {
this.get = doc.data().lastTime;
} else {
this.get = 'Never done';
}
}).catch(error => {
console.log('Error getting document:', error);
});
return this.get;
}
For my test I actually have a string value inside the doc 'lastTime' which is a string.
Inside ngOnInit(), I called my function and console.log the result
this.InjuredLastTime = this.getLastTime('INJURY');
console.log(this. this.InjuredLastTime);
Normally I should have my string printed inside the console but I got undefined...
Maybe it's because Firestore do not fetch fast enough my data, but I am quiet surprised since Firestore is quiet fast normally...

You don't actually wait for the promise that is created by docRef.get() before you return from getLastTime(). So, unless the call to firebase is instant (e.g. never) this won't work.
The correct solution really depends on what you are doing with this.InjuredLastTime. But one approach would just be to return a promise and set it after it is ready:
getLastTime(collectionName: string) {
const docRef = this.afs.firestore.collection(collectionName).doc(this.User).collection('lastTime').doc('lastTime');
return docRef.get().then(doc => {
if (doc.exists) {
return doc.data().lastTime;
} else {
return 'Never done';
}
}).catch(error => {
console.log('Error getting document:', error);
return null;
});
}
Then, instead of the assignment synchronously, do it asynchronously:
this.getLastTime('INJURY').then(result => { this.InjuredLastTime = result });

Data is loaded from Firestore asynchronously, since it may take some time before the data comes back from the server. To prevent having to block the browser, your code is instead allowed to continue to run, and then your callback is called when the data is available.
You can easily see this with a few well-placed log statements:
const docRef = this.afs.firestore.collection(collectionName).doc(this.User).collection('lastTime').doc('lastTime');
console.log('Before starting to get data');
docRef.get().then(doc => {
console.log('Got data');
});
console.log('After starting to get data');
If you run this code, you'll get:
Before starting to get data
After starting to get data
Got data
This is probably not the order that you expected the logging output in, but it is actually the correct behavior. And it completely explains why you're getting undefined out of your getLastTime function: by the time return this.get; runs, the data hasn't loaded yet.
The simplest solution in modern JavaScript is to mark your function as async and then await its result. That would look something like this:
async function getLastTime(collectionName: string) {
const docRef = this.afs.firestore.collection(collectionName).doc(this.User).collection('lastTime').doc('lastTime');
doc = await docRef.get();
if (doc.exists) {
this.get = doc.data().lastTime;
} else {
this.get = 'Never done';
}
return this.get;
}
And then call it with:
this.InjuredLastTime = await this.getLastTime('INJURY');

Wait till await/async returns in Jquery

I am new to await/async in Jquery. When I am trying to use it in my js file, It is getting executed, but not as expected.
async updateProduct (product) {
product.travelers.forEach((traveler, travelerOrder) => this.updateProductwithPassengerName(traveler) )
return product
}
async updateProductwithPassengerName (traveler) {
const travelerId = traveler.profile_id;
var body = await this.userServiceClient.request('/v2/users/' + travelerId + '/info','GET')
traveler.first_name = body.first_name
traveler.last_name = body.last_name
return traveler
}
async request (path, method, body) {
const options = {
method: method,
body: body,
headers: this.headers(),
credentials: 'same-origin'
}
const response = await fetch(this.relativeUrl(path), options)
if (!response.ok) {
throw new HttpError('Technical error occured', response.status)
}
return response.json().catch(function () {
throw new HttpError('No results found', response.status)
})
}
Those are the 3 functions. What is happening now is that
traveler.first_name = body.first_name
traveler.last_name = body.last_name
these are not setting in synchronous manner( after
var body = await this.userServiceClient.request('/v2/users/' +
travelerId + '/info','GET')
. These are executing after a quiet a long time.
What I am doing here is that, for each traveler I am setting first_name and last_name. And updating the product object. Since this setting of values is happening after a long time, product object is getting updated later, by the time UI page renders. I want this setting values to happening before doing anything else in jquery.
Looking for help

The problem is that forEach will not await the asynchronous result and so your first function returns a promise that immediately resolves, without waiting for the requests to finish.
Here is how to correct:
async updateProduct (product) {
await Promise.all(product.travelers.map((traveler, travelerOrder) => this.updateProductwithPassengerName(traveler) ));
return product
}
Or, if the backbone cannot deal with all these concurrent requests, then wait for each of them to complete before issuing the next:
async updateProduct (product) {
for (let traveler of product.travelers) {
await this.updateProductwithPassengerName(traveler);
}
return product
}

when using asyn/await in JQuery or angularjs or nodejs, use promise. Promise will complete the process and then return either success or fail. Then only execute next steps.Go through below links and you will get idea.
Modern Asynchronous JavaScript with Async and Await
async & await in Javascript