In my project (VUE + Vuex) I need to make some API requests simultaneously, according to some contents and then process the results.
The getters.api_props(key) function will return the method ('post', 'patch', 'delete') or false if there is no need for a request. It will also return the url and the object that is needed for the request.
The api method returns the request as a Promise using axios.
Here is my code so far:
var contents = {person: {...}, info: {...}}
var promiseArray = [];
for (var key in contents) {
let [method, url, hash] = getters.api_props(key);
if (method) { promiseArray.push(api[method](url, hash)) }
}
await Promise.allSettled(promiseArray).then((results) => {
results.map(r => {
// THE RESULTS WILL BE PROCESSED HERE like:
// commit("save", [key, r])
console.info(r)
})
}).catch(e => console.log('ERROR:::',e)).finally(commit("backup"))
The problem is that the results does not include the 'key' so the save method that is called cannot know where to save the results.
Can you propose a fix or a better solution?
I would recommend to write
const contents = {person: {...}, info: {...}}
cosnt promiseArray = [];
for (const key in contents) {
let [method, url, hash] = getters.api_props(key);
if (method) {
promiseArray.push(api[method](url, hash)).then(value => ({
key,
status: 'fulfilled',
value
}), reason => ({
key,
status: 'rejected',
reason
})))
}
}
const results = await Promise.all(promiseArray);
for (const r of results) {
if (r.status=='fulfilled') {
console.info(r.key, ':', r.value.data)
commit("save", [r.key, r.value]);
} else if (r.status=='rejected') {
console.warn(r.key, ':', r.reason)
}
})
commit("backup");
So, to answer my own question, after Bergi's comments I filled promiseArray with
api[method](url, hash).then((r) => [key, r]).catch((e) => {throw [key, e.response]})
and then found the key that I needed:
await Promise.allSettled(promiseArray).then((results) => {
results.map((r) => {
if (r.status=='fulfilled') {
console.info(r.value[0],':',r.value[1].data)
}
if (r.status=='rejected') {
console.warn(r.reason[0],':',r.reason[1])
}
})
})
You obviously don't need to take this, but I fiddled with it for a while and this is what I liked best:
import forEach from 'lodash/forEach'
import mapValues from 'lodash/mapValues'
import { api, getters } from 'somewhere'
var contents = {person: {...}, info: {...}}
const promiseContents = mapValues(contents, (value, key) => {
let [method, url, hash] = getters.api_props(key);
if (!method) { return }
return api[method](url, hash)
})
await Promise.allSettled(Object.values(promiseContents))
forEach(promiseContents, (promise, key) => {
promise.then(response => {
if (promise.status === 'rejected') {
console.warn(key, ':', response)
}
console.info(key, ':', value.data)
})
})
The big requirement is that you include lodash in the project, but that is not an unusual ask in javascript projects.
mapValues allows you to keep the structure of your contents object while replacing the values with promises. I just use await on Promise.allSettled to tell the rest of the code when it can go. I just ignore the results.
Finally, using lodash's forEach I interpret the results. The advantage here is that every promise is run in a function alongside the key from your original contents object.
I like doing it this way because it doesn't require you to create a [key, result] array. That said, either way works fine.
Related
I'm creating a JS function that will make a call to an API, loop through the returned data and perform another call to retrieve more information about the initial data (for example where the first call return an ID, the second call would return the name/address/number the ID corresponds to). Positioning the async and await keywords though, have proven to be way more challenging than I imagined:
useEffect(() => {
const getAppointments = async () => {
try {
const { data } = await fetchContext.authAxios.get('/appointments/' + auth.authState.id);
const updatedData = await data.map(value => {
const { data } = fetchContext.authAxios.get('/customerID/' + value.customerID);
return {
...value, // de-structuring
customerID: data
}
}
);
setAppointments(updatedData);
} catch (err) {
console.log(err);
}
};
getAppointments();
}, [fetchContext]);
Everything get displayed besides the customerID, that results undefined. I tried to position and add the async/await keywords in different places, nothing works. What am I missing?
map returns an array, not a promise. You need to get an array of promises and then solve it (also, if your way worked, it would be inefficient waitting for a request to then start the next one.)
const promises = data.map(async (value) => {
const { data } = await fetchContext.authAxios.get('/customerID/' + value.customerID);
return {
...value,
customerID: data
};
});
const updatedData = await Promise.all(promises);
I'm working on a local Node.js app that needs to access the Google Chrome cookies. I've found the chrome-cookies-secure library that seems to do the job but I just can't figure out what's wrong with the code below.
const chrome = require('chrome-cookies-secure');
const domains = [
"google.com"
];
const resolveCookies = async () => {
let result = [];
for(domain of domains) {
await chrome.getCookies(`https://${domain}/`, (err, cookies) => {
result.push(cookies);
// console.log(cookies); //? This is going to correctly print the results
})
}
return result;
}
const final = resolveCookies();
console.log(final); //! This is going to return a Promise { <pending> } object
The idea is that I just want to store the cookies from all the domains in a list but no matter what I cannot resolve the Promise.
I didn't see any examples with the async call for this module but if I don't use it it's going to return me an empty list after the script execution.
My Node Version: v14.4.0
What am I doing wrong?
It looks like the implementation of getCookies is not correctly awaiting the asynchronous processes. You can see in the implementation that although getCookies itself is async, it calls getDerivedKey without awaiting it (and that function isn't async anyway).
Rather than trying to rely on this implementation, I'd suggest using Util.promisify to create a proper promise via the callback:
const util = require('util');
const chrome = require('chrome-cookies-secure');
const getCookies = util.promisify(chrome.getCookies);
// ...
const cookies = await getCookies(`https://${domain}/`);
Note that, as Reece Daniels pointed out in the comments, the getCookies implementation actually takes a profile parameter after the callback; if you need to use that parameter, you can't use the built-in promisify. You'd have to wrap it yourself instead, this could look like e.g.:
const getCookies = (url, format, profile) => new Promise((resolve, reject) => {
chrome.getCookies(url, format, (err, cookies) => {
if (err) {
reject(err);
} else {
resolve(cookies);
}
}, profile);
});
They already tried to fix the promise upstream, but the PR hasn't been merged in nearly nine months.
Note that once you have a working function to call you can also convert:
const resolveCookies = async () => {
let result = [];
for(domain of domains) {
await chrome.getCookies(`https://${domain}/`, (err, cookies) => {
result.push(cookies);
// console.log(cookies); //? This is going to correctly print the results
})
}
return result;
}
to simply:
const resolveCookies = () => Promise.all(domains.map((domain) => getCookies(`https://${domain}/`)));
An async function returns a Promise.
So your resolveCookies function will also return a Promise as you noticed.
You need to either chain the console.log with a .then e.g.
resolveCookies().then(console.log);
Or if you need to set it to a variable like final you need to await that Promise too. In that case you need an async IIFE:
(async () => {
const final = await resolveCookies();
console.log(final);
})();
try this.
const chrome = require('chrome-cookies-secure');
const domains = [
"www.google.com"
];
const resolveCookies = async() => {
let result = [];
for (domain of domains) {
const cookies = await getCookies(domain)
result.push(cookies)
}
return Promise.resolve(result);
}
const getCookies = async (domain) => {
chrome.getCookies(`https://${domain}/`, (err, cookies) => {
return Promise.resolve(cookies);
})
}
resolveCookies().then((resp) => {
console.log('FINAL ',resp)
}).catch((e) => {
console.log('ERROR ', e)
})
I am trying to filter my array using Array.filter() function, however I came across this issue.
I need to call some other function inside the filter function asynchronously. However the array is not changing its value based on the conditions that I define in the function.
const filterWithEmail = data.filter(async (doc) =>
{
const findUser = await UserService.findUser(doc.id).catch(err => {});
if (findUser)
{
const { email } = findUser;
return regexFilter ? regexFilter.test(email.normalize("NFKC")) : false;
}
});
This code doesn't affect the data array at all for some reason.
Any suggestions what I'm doing wrong?
Thank you in advance.
filter expects the return value of the callback to be a boolean but async functions always return a promise.
You don't know if you want to return true or false in time to tell filter which it is.
What you possibly want to do is:
map the data from data to { keep: true, original_data: data } (using an async callback)
Pass the resulting array of promises to Promise.all
await the return value of Promise.all
filter that array with: .filter(data => data.keep)
Get the original objects back with .map(data => data.original_data)
Something along these lines (untested):
const filterWithEmail = (
await Promise.all(
data.map(async (data) => {
const findUser = await UserService.findUser(doc.id).catch((err) => {});
let keep = false;
if (findUser && regexFilter)
keep = regexFilter.test(email.normalize("NFKC"));
return { data, keep };
})
)
)
.filter((data) => data.keep)
.map((data) => data.data);
I have an array where I need to call an API endpoint for each index. Once that is resolved I need to append it in that element. I want to return the updated array once this gets completed for each index of the array.
I tried using async-await in this way
// Let input be [{'x': 1, 'y': 2}, {'x': 11, 'y': 22}, ...]
async function hello(input) {
await input.forEach(element => {
fetch(url, options)
.then((res) => {
element['z'] = res
})
})
return input
}
I need to use this function to update my state
hello(data)
.then((res: any) => {
this.setState((prevState) => ({
...prevState,
inputData: res,
}))
})
The issue is that I need one more forced render for key 'z' to show.
How to resolve this?
I don't have much experience using async await so I am not sure if I am using it correctly.
The correct way is to use Promise.all and return the promise to be used by the caller function since you want the entire updated input value to be set in state.
In your case forEach doesn't return a promise so await on it is useless.
Also if you use await within the forEach function, you need to be able provide away to let the hello function's .then method call when all promises have resolved. Promise.all does that for you
function hello(input) {
const promises = [];
input.forEach(element => {
promises.push(
fetch(url, options)
.then(res => res.json()
.then((result) => {
// return the updated object
return {...element, z: result};
})
)
});
return Promise.all(promises);
}
...
hello(data)
.then((res: any) => {
this.setState((prevState) => ({
...prevState,
inputData: res,
}))
})
P.S. Note that the response from fetch also will need to be called with res.json()
async/await won't work in loops which uses callback(forEach, map, etc...)
You can achieve your result using for..of loop.
Try this and let me know if it works.
function getResult() {
return new Promise((resolve) => {
fetch(url, options)
.then((res) => {
return resolve(res);
})
})
}
async function hello(input) {
for (let element of input) {
let res = await getResult(element);
element['z'] = res;
}
}
I'm trying to only make one http call at time but when I log the response from getUrl they are piling up and I start to get 409s (Too many requests)
function getUrl(url, i, cb) {
const fetchUrl = `https://api.scraperapi.com?api_key=xxx&url=${url.url}`;
fetch(fetchUrl).then(async res => {
console.log(fetchUrl, 'fetched!');
if (!res.ok) {
const err = await res.text();
throw err.message || res.statusText;
}
url.data = await res.text();
cb(url);
});
}
let requests = urls.map((url, i) => {
return new Promise(resolve => {
getUrl(url, i, resolve);
});
});
const all = await requests.reduce((promiseChain, currentTask) => {
return promiseChain.then(chainResults =>
currentTask.then(currentResult => [...chainResults, currentResult]),
);
}, Promise.resolve([]));
Basically I don't want the next http to start until the previous one has finished. Otherwise I hammer their server.
BONUS POINTS: Make this work with 5 at a time in parallel.
Since you're using await, it would be a lot easier to use that everywhere instead of using confusing .thens with reduce. It'd also be good to avoid the explicit Promise construction antipattern. This should do what you want:
const results = [];
for (const url of urls) {
const response = await fetch(url);
if (!response.ok) {
throw new Error(response); // or whatever logic you need with errors
}
results.push(await response.text());
}
Then your results variable will contain an array of response texts (or an error will have been thrown, and the code won't reach the bottom).
The syntax for an async function is an async keyword before the argument list, just like you're doing in your original code:
const fn = async () => {
const results = [];
for (const url of urls) {
const response = await fetch(url);
if (!response.ok) {
throw new Error(response); // or whatever logic you need with errors
}
results.push(await response.text());
}
// do something with results
};
To have a limited number of requests at a time, make a queue system - when a request completes, recursively call a function that makes another request, something like:
const results = [];
const queueNext = async () => {
if (!urls.length) return;
const url = urls.shift();
const response = await fetch(url);
if (!response.ok) {
throw new Error(response); // or whatever logic you need with errors
}
results.push(await response.text());
await queueNext();
}
await Promise.all(Array.from({ length: 5 }, queueNext));
// do something with results
You cannot use Array methods to sequentually run async operations because array methods are all synchronous.
The easiest way to achieve sequential async tasks is through a loop. Otherwise, you will need to write a custom function to imitate a loop and run .then after a async task ends, which is quite troublesome and unnecessary.
Also, fetch is already returning a Promise, so you don't have to create a Promise yourself to contain that promise returned by fetch.
The code below is a working example, with small changes to your original code (see comments).
// Fake urls for example purpose
const urls = [{ url: 'abc' }, { url: 'def', }, { url: 'ghi' }];
// To imitate actual fetching
const fetch = (url) => new Promise(resolve => {
setTimeout(() => {
resolve({
ok: true,
text: () => new Promise(res => setTimeout(() => res(url), 500))
});
}, 1000);
});
function getUrl(url, i, cb) {
const fetchUrl = `https://api.scraperapi.com?api_key=xxx&url=${url.url}`;
return fetch(fetchUrl).then(async res => { // <-- changes here
console.log(fetchUrl, 'fetched!');
if (!res.ok) {
const err = await res.text();
throw err.message || res.statusText;
}
url.data = await res.text();
return url; // <--- changes here
});
}
async function getAllUrls(urls){
const result = [];
for (const url of urls){
const response = await getUrl(url);
result.push(response);
}
return result;
}
getAllUrls(urls)
.then(console.log);
async/await is perfect for this.
Assuming you have an array of URLs as strings:
let urls = ["https://example.org/", "https://google.com/", "https://stackoverflow.com/"];
You simply need to do:
for (let u of urls) {
await fetch(u).then(res => {
// Handle response
}).catch(e => {
// Handle error
});
}
The loop will not iterate until the current fetch() has resolved, which will serialise things.
The reason array.map doesn't work is as follows:
async function doFetch(url) {
return await fetch(url).then(res => {
// Handle response
}).catch(e => {
// Handle error
});
}
let mapped = urls.map(doFetch);
is equivalent to:
let mapped;
for (u of urls) {
mapped.push(doFetch(u));
}
This will populate mapped with a bunch of Promises immediately, which is not what you want. The following is what you want:
let mapped;
for (u of urls) {
mapped.push(await doFetch(u));
}
But this is not what array.map() does. Therefore using an explicit for loop is necessary.
Many people provided answers using for loop. But in some situation await in for loop is not welcome, for example, if you are using Airbnb style guide.
Here is a solution using recursion.
// Fake urls for example purpose
const urls = [{ url: 'abc' }, { url: 'def', }, { url: 'ghi' }];
async function serialFetch(urls) {
return await doSerialRecursion(
async (url) => {
return result = await fetch(url)
.then((response) => {
// handle response
})
.catch((err) => {
// handle error
});
},
urls,
0
);
}
async function doSerialRecursion(fn, array, startIndex) {
if (!array[startIndex]) return [];
const currResult = await fn(array[startIndex]);
return [currResult, ...(await doSerialRecursion(array, fn, startIndex + 1))];
}
const yourResult = await serialFetch(urls);
The doSerialRecursion function will serially execute the function you passed in, which is fetch(url) in this example.