We Keep Coding

How To Merge Parallel Axios GET Requests and Promise allSettled Function With Source Objects

I can successfully resolve parallel execution of Axios GET requests using the allSettled Promise method. However, I can not merge the keys of the source object that contains the URL with the results of the GET request.
So, I have an array of objects that contains the URL I want to perform GET requests on, as well as some other info that I want to keep. This looks like the following:
const axios = require('axios');
const sources =[
{
name: 'Source1',
url: 'https://run.mocky.io/v3/15ce0aa4-96b4-4af7-9252-b66b271cd73f'
},
{
name: 'Source2',
url: 'https://run.mocky.io/v3/a5a46911-3ad0-4342-a1a1-f753ea3f2241'
}];
async function run() {
const tasks = sources.map(source => axios.get(source.url));
const results = await Promise.allSettled(tasks);
const fulfilled = results.filter(result => result.status === 'fulfilled');
console.log(fulfilled);
}
run();
Running the above code the fulfilled variable will be an array of fulfilled Promise outcome objects. That object looks approximately like:
{
status: 'fulfilled',
value: {
status: 200,
statusText: 'OK',
headers: [Object],
config: [Object],
request: [ClientRequest],
data: ‘DATA FROM THE AXIOS GET REQUEST IS ADDED HERE’
},
I want to include the ‘name’ key and value pair from my original sources array objects in each fulfilled promise object.
So, to be clear I want the following to be returned (notice the ‘name’ key’).
{
status: 'fulfilled',
value: {
status: 200,
statusText: 'OK',
headers: [Object],
config: [Object],
request: [ClientRequest],
data: ‘DATA FROM THE AXIOS GET REQUEST IS ADDED HERE’,
name: ‘Source1’ // I want this added from the original source object.
},
I have not found a way to achieve this prior to the allSettled method being reached. Ideally, I’d like to do this inside the map function. However, if I return an object with a key with its value set as the axios GET request, it does not resolve and remains in pending status even after allSettled marks the overall request as fulfilled.
const tasks = sources.map(source => axios.get(source.url));
Can anyone help?

Instead of passing only source.url, pass the whole source object. Axios will take this as its config object.
Then, you will find this object copied in response.config.
async function run() {
const tasks = sources.map(axios.get); // Passing the whole { url : "..." , name : "Source2" } as "config". This syntax is equivalent to sources.map(source => axios.get(sources))
const responses = await Promise.allSettled(tasks);
let results = responses.map( response => {
let result = response.data;
result.name = response.config.name; // <-- here
return result;
})
const fulfilled = results.filter(result => result.status === 'fulfilled');
console.log(fulfilled);
}
EDIT
I'm not 100% sure about the axios syntax though, according to the doc, it can be axios(config) or axios.get(url, config). You can try const tasks = sources.map(axios); or const tasks = sources.map(source => axios.get(sources.url, source));

I want to check whether URL exists or not in Synchronous way in javascript NODE.js

Guys i am new to the node js
Thank u guys
🙌🙏.
const urlExist = require("url-exist");
var httpUrl='';
for (var i=0; i<req.body.url.length; i++) {
httpUrl = req.body.url[i];
let exists1 = () => Promise.resolve(urlExist(httpUrl),httpUrl);
Promise.all([exists1()]).then((resultArr)=>{
console.log(resultArr[0],httpUrl)
})
}
The input is given as a array from POSTMAN
{
"urls":["http://www.wrong.com/","http://wrong.com/","http://example.com","http://example.com","http:/example.com","http:/example5.com"]
}
This the OUTPUT i am getting
true http:/example5.com
false http:/example5.com
true http://example5.com
true http://example5.com
true http://example5.com
false http://example5.com
The EXPECTED output is:-
false http://www.wrong.com/
false http://wrong.com/
true http:/example.com
true http:/example.com
true http:/example.com
true http:/example5.com
But the thing is this function reading only "HTTP:/example5.com" and not giving the same Boolean value
Thank you guys

You don't need to wrap it in promise and then use a promise all inside the loop. Simply doing this should work
const urlExist = require("url-exist");
const urls = ["http://www.wrong.com/","http://wrong.com/","http://example.com","http://example.com","http:/example.com","http:/example5.com"];
Promise.all(urls.map(async url => {
const result = await urlExist(url);
return {"status": result, url};
}))
.then(res => console.log(res));
You can modify the output the way you want. At the moment it is logging it as an array

Also, please note that using Promise.all to execute multiple queries to the same url here is done simultaneously, hence maybe the weird result (server might refuse when being sent too many requests from the same IP for example).
If you want to avoid it, you could use for ... of loop which will execute each request one after another sequentially, like for example :
const exists = require('url-exist')
const urls = [
'http://www.wrong.com/',
'http://wrong.com/',
'http://example.com',
'http://example.com',
'http:/example.com',
'http:/example5.com'
]
const run = async () => {
const results = []
for (const url of urls) {
try {
const result = await exists(url)
results.push({ result, url })
} catch (e) { console.error(`error with ${url} : ${e}`) }
}
return results
}
run()
.then(console.log)
.catch(console.error)
Hope this helps! :)

Returning Response object with JSON from service worker

I am trying to hook into the fetch request for some JSON and check if the data is in IndexedDB to get it before going to the network. Something odd is happening with my promises not resolving I think.
Here is my code:
event.respondWith(async function() {
var data = {success:true, msg:'', data: []};
localforage.iterate(function(value, key) {
data.data.push([key, value])
})
if (data.data.length > 0) {
return await new Response(JSON.stringify(data),{
headers: { "Content-Type" : "application/json" }
});
}
let response = await fetch(event.request)
// clone response as it can only be used once and needs to be returned for client fetch
let responseData = await response.clone().json()
await responseData.data.forEach(function (todo) {
localforage.setItem(todo[0], todo[1])
})
return response
}())
However, my front end gets an empty response object when the if resolves true (data in IndexedDB). It's as if my response object is being created before the IndexedDB promise resolves.

It's a bit of a guess without being able to run the code but
I think you need to wait for localforage.iterate since it returns a
promise
I think you also need to wait for localforage.setItem since
it returns a promise too - using Promise.all and map should handle that for you
var data = {success:true, msg:'', data: []};
await localforage.iterate((value, key) => {
data.data.push([key, value])
})
if (data.data.length > 0) {
return new Response(JSON.stringify(data),{
headers: { "Content-Type" : "application/json" }
});
}
let response = await fetch(event.request)
// clone response as it can only be used once and needs to be returned for client fetch
let responseData = await response.clone().json()
await Promise.all(responseData.data.map(todo => {
return localforage.setItem(todo[0], todo[1]);
}));
return response;

Using Bluebird Promise in For Loop to build and return an Array of Objects

I am trying to build and return an Object using Bluebird Promises.
The Promise is a HTTP request which gets additional data to add to the Object.
I created a function that carries out a request in a for loop (I am also using a framework that carries out some middleware - this is what the z. is about)
const getWebAppCustomFieldDetails = (z, url) => {
const responsePromise = z.request({
url:url,
headers:{
'content-type': 'application/json'
}
});
return responsePromise
.then(response =>{
return JSON.parse(response.content);
});
};
This function is called within the following code:
const webAppFields = (z, bundle) => {
//This section carries creates an initial request which gets the bulk of the data
const responsePromise = z.request({
url: webAppUrl(bundle) + '/' + encodeURI(bundle.inputData.webApp),
headers:{
'content-type': 'application/json'
},
});
//This is an array to hold the objects created from the response
var fields = [];
return responsePromise
.then(response => {
response = JSON.parse(response.content);
//From the response, append the core fields
response.systemFields.forEach( function (systemField) {
fields.push({
'key': systemField.name,
'required': systemField.required,
'type': systemField.type.toLowerCase()
});
});
return response;
})
.then(response => {
//Sometimes there are custom fields that need to be retrieved individually
const customFieldCount = response.fields.length;
var customFieldAppend = '';
for (var i = 0; i < customFieldCount; i++){
getWebAppCustomFieldDetails(z, response.fields[0].links[0].uri)
.then(response =>{
customFieldAppend = {
'key': response.name,
'required': response.required,
'type': response.type.toLowerCase()
};
//This push doesn't updated the fields array!
fields.push(customFieldAppend);
});
}
//This return does not include the custom fields!
return fields;
});
};
I cannot figure out how to return the value from the nested Promise

You can use Promise.reduce to reduce the list of promises you are creating inside the for loop into a single promise:
...
.then(response => {
const customFieldCount = response.fields.length;
var customFieldAppend = '';
// Promice.reduce will return the accumulator "totalFields"
return Promise.reduce(response.fields, (totalFields, field) => {
getWebAppCustomFieldDetails(z, field.links[0].uri) // Using the "field" variable provided by the reducer function
.then(response => {
customFieldAppend = {
'key': response.name,
'required': response.required,
'type': response.type.toLowerCase()
};
// Add the data to the accumulator "totalFields"
totalFields.push(customFieldAppend);
});
}, []); // The third argument is the initial value of the accummulator. In your example it is a an array, so the initial value is an empty array.
});
...
Promise.reduce takes in input three arguments: the list of arguments to loop on (response.fields), the reducer function and the initial value for the accumulator. The accumulator (which I called totalFields) is the first argument of the reducer function, it is the variable used to reduce the values in the list in a single value. In your case the accumulator is an array (the fields array used in your example) so its initial value is an empty array.
Inside the reduce function you can access the single elements of the list (second argument) and you can call your async operation and fill the accumulator ad every step. The Promise.reduce function will return the accumulator wrapped inside a promise. For this reason your function can directly returns the Promise.reduce's returned promise.

How to loop request-promise API request call?

I am learning Node.JS and I am introduced to request-promise package. I use it for API call but I faced a problem where I cannot apply loop to it.
This is the example showing a simple API call:
var read_match_id = {
uri: 'https://api.steampowered.com/IDOTA2Match_570/GetMatchHistory/V001',
qs: {
match_id: "123",
key: 'XXXXXXXX'
},
json: true
};
rp(read_match_id)
.then(function (htmlString) {
// Process html...
})
.catch(function (err) {
// Crawling failed...
});
How can i have loop like this:
var match_details[];
for (i = 0; i < 5; i++) {
var read_match_details = {
uri: 'https://api.steampowered.com/IDOTA2Match_570/GetMatchDetails/V001',
qs: {
key: 'XXXXXXXXX',
match_id: match_id[i]
},
json: true // Automatically parses the JSON string in the response
};
rp(read_match_details)
.then (function(read_match){
match_details.push(read_match)//push every result to the array
}).catch(function(err) {
console.log('error');
});
}
And how can I know when all the async request are done?

request-promise uses Bluebird for Promise.
The simple solution is Promise.all(ps), where ps is array of promises.
var ps = [];
for (var i = 0; i < 5; i++) {
var read_match_details = {
uri: 'https://api.steampowered.com/IDOTA2Match_570/GetMatchDetails/V001',
qs: {
key: 'XXXXXXXXX',
match_id: match_id[i]
},
json: true // Automatically parses the JSON string in the response
};
ps.push(rp(read_match_details));
}
Promise.all(ps)
.then((results) => {
console.log(results); // Result of all resolve as an array
}).catch(err => console.log(err)); // First rejected promise
The only disadvantage of this is, this will go to catch block immediately after any of the promise is rejected. 4/5 resolved, won't matter, 1 rejected will throw it all to catch.
Alternative approach is to use Bluebird's inspection (refer this). We'll map all promises to their reflection, we can do an if/else analysis for each promise, and it will work even if any of the promise is rejected.
// After loop
ps = ps.map((promise) => promise.reflect());
Promise.all(ps)
.each(pInspection => {
if (pInspection.isFulfilled()) {
match_details.push(pInspection.value())
} else {
console.log(pInspection.reason());
}
})
.then(() => callback(match_details)); // Or however you want to proceed
Hope this will solve your problem.