I'm currently using Promise.allSettled and am mutating a successful response. However, I cannot figure out how to mutate the error response. It will always return an object for each promise of:
{ status: error, reason: reasonObject }
I would like to add two properties to the object so I can track the IDs and other meta data.
This is how I'm currently adding two keys to a successful response:
const results = await Promise.allSettled(
batchedPayloads.map(async (payload, idx) => {
return { accountId: payload && payload?.accountId, tag: payload?.tagId, result: await mutation(payload) };
})
I would like to do the same to an error response, while still using async/await
Add a .catch onto the call of mutation so that if there's an error, it returns the desired object structure. Do this inside the allSettled loop. To be concise, define the object of properties to be added in advance.
const results = await Promise.allSettled(
batchedPayloads.map(payload => {
// properties to add
const obj = { accountId: payload && payload?.accountId, tag: payload?.tagId };
return mutation(payload)
.then(result => ({ result, ...obj }))
.catch((error) => {
Object.assign(error, obj);
// throw here so that allSettled shows this in the error section
throw error;
})
})
);
Related
I have a Firebase function that is attempting to take an array of UIDs and return an array of User Objects. I am trying to use Promise.all() to return all of the async results, but I am getting an empty array returned. I am, however, getting the logged out results after the fact.
const fetchUserObjects = function(uids){
let promises = []
uids.forEach((uid) => {
admin.database().ref(`/users/${uid}`).once("value")
.then(function(dataSnapshot) {
const userDataAll = dataSnapshot.val()
const userData = {}
userData.id = userDataAll.id
userData.username = userDataAll.username
userData.first = userDataAll.first
userData.last = userDataAll.last
promises.push(userData)
console.log(userData)
})
.catch((error) => {
// Re-throwing the error as an HttpsError so that the client gets the error details.
throw new functions.https.HttpsError('unknown', error.message, error);
});
})
return Promise.all(promises);
}
return fetchUserObjects(uids)
fetchUserObjects is always returning an empty array. Nothing is ensuring that the async work started by once() is copmlete before you push values into the array. Also notice that you're not actually pushing promises into that array. You're pushing plain old JavaScript objects. You need to push actual promises into the array instead, and you need to do it without having to wait for other promises to resolve. Here's what it should look like instead:
const fetchUserObjects = function(uids){
let promises = []
uids.forEach((uid) => {
const promise = admin.database().ref(`/users/${uid}`).once("value")
.then(function(dataSnapshot) {
const userDataAll = dataSnapshot.val()
const userData = {}
userData.id = userDataAll.id
userData.username = userDataAll.username
userData.first = userDataAll.first
userData.last = userDataAll.last
console.log(userData)
return userData
})
.catch((error) => {
// Re-throwing the error as an HttpsError so that the client gets the error details.
throw new functions.https.HttpsError('unknown', error.message, error);
});
promises.push(promise)
})
return Promise.all(promises);
}
Notice that a promise is immediately pushed into the promises array, and it will resolve with a userData object that was returned by the then callback.
I am doing a simple api get request, but I can't seem to isolate the array on its own. its always inside of a promise and I'm not sure how to remove it or how to access the values stored in the array.
function getLocation(name) {
let output = fetch(`http://dataservice.accuweather.com/locations/v1/cities/search?apikey=oqAor7Al7Fkcj7AudulUkk5WGoySmEu7&q=london`).then(data => data.json());
return output
}
function App() {
var output = getLocation(`london`);
console.log (output)
...
__proto__: Promise
[[PromiseStatus]]: "resolved"
[[PromiseValue]]: Array(3)
is what is displayed in the console.log I require just the Array(3)
fetch, and Promise#then, always return a promise. To access the information it fetches, consume the promise:
getLocation()
.then(data => {
// use the data
})
.catch(error => {
// Handle/report the error
});
or in an async function:
const data = await getLocation();
Side note: Your getLocation has a very common error (so common I wrote it up on my anemic little blog): It doesn't check that the HTTP operation succeeded. fetch only fails on network errors, not HTTP errors. To fix it:
function getLocation(name) {
return fetch(`http://dataservice.accuweather.com/locations/v1/cities/search?apikey=oqAor7Al7Fkcj7AudulUkk5WGoySmEu7&q=london`)
.then(response => {
if (!response.ok) {
throw new Error("HTTP error, status = " + response.status);
}
return response.json();
});
}
I wrote a snippet of code that gets some a JSON from the Foursquare API. From this JSON, I get IDs of venues. These IDs are then used to get more details from those specific venues by issuing a fetch() request for every ID, and mapping those requests in an array. That array is then passed into Promise.all(). When the API is available, everything works, but it's the error catching that I can't get my head around.
fetch(`https://api.foursquare.com/v2/venues/search?${params}`)
.then(response => response.json())
.then(data => {
const venueIds = data.response.venues.map(venue => venue.id)
const venuePromises = venueIds.map(venueId => {
fetch(`https://api.foursquare.com/v2/venues/${venueId}?${otherParams}`)
.then(response => {
// Must check for response.ok, because
// catch() does not catch 429
if (response.ok) {
console.log('ok')
return response.json()
} else {
Promise.reject('Error when getting venue details')
}
})
})
Promise.all(venuePromises).then(data => {
const venues = data.map(entry => entry.response.venue) // Error for this line
this.parseFsqData(venues)
}).catch((e) => {console.log(e); getBackupData()})
}).catch((e) => {console.log(e); getBackupData()})
function getBackupData() {
console.log('backup')
}
When the API is not available, I get the following console errors (and more of the same):
TypeError: Cannot read property 'response' of undefined
at MapsApp.js:97
at Array.map (<anonymous>)
at MapsApp.js:97
backup
api.foursquare.com/v2/venues/4b7efa2ef964a520c90d30e3?client_id=ANDGBLDVCRISN1JNRWNLLTDNGTBNB2I4SZT4ZQYKPTY3PDNP&client_secret=QNVYZRG0JYJR3G45SP3RTOTQK0SLQSNTDCYXOBWUUYCGKPJX&v=20180323:1 Failed to load resource: the server responded with a status of 429 ()
Uncaught (in promise) Error when getting venue details
I don't understand why then() after Promise.all() is entered, because response is never ok (there is no ok logged in console). Also, I don't understand why the console.log()'s in the catch() blocks aren't executed, or why they are empty. I don't see any caught error information in console, but still the getBackupData function is called. Finally, it is unclear why the last message in console indicates that the error is uncaught, as I expected reject() to make Promise.all() fail.
How can I tactfully catch any errors (included those not normally caught by catch(), such as 429 errors) and call getBackupData when any errors occur?
Your issues are related: namely, the Promise chain must be returned. If you do not return the Promise, you disconnect any of the caller's Promise#catch handling, and any errors in your Promise / then code will result in unhandled promise rejection errors, such as what you have obtained:
Uncaught (in promise) Error when getting venue details
This uncaught promise rejection appears in your code that handles the resolution of fetch:
if (response.ok) {
console.log('ok')
return response.json()
} else {
Promise.reject('Error when getting venue details') // <----
}
Since this code is being used to construct your venuePromises array, its return value will populate venuePromises. If the response was ok, that array element will have the response JSON from return response.json(). If the response failed, there is no return statement that executes, so the array element has the value undefined. Thus, venuePromises would look like this:
[
{ /** some object for successful response */ },
undefined,
{ /** some other object */ },
...
]
Thus when this array is accessed by your Promise.all's success handler, you get the TypeError since you expected all elements of venuePromises to be valid. This TypeError is caught by the Promise.all's .catch handler (which is why it is logged, and you receive the "backup" text in your log).
To fix, you need to return the Promise.reject, and also the Promise.all. Note that there are some cases of implicit return, but I find it nicer to be explicit, especially if the statement spans multiple lines. Since you're returning the Promise.all statement, you can offload its .then and .catch to the caller, resulting in one less nesting level, and one fewer duplicated .catch handler.
fetch(`https://api.foursquare.com/v2/venues/search?${params}`)
.then(response => response.json())
.then(jsonData => {
const venueIds = jsonData.response.venues.map(venue => venue.id);
const venuePromises = venueIds.map(venueId => {
let link = `https://api.foursquare.com/v2/venues/${venueId}?${otherParams}`;
return fetch(link).then(response => {
// Must check for response.ok, because catch() does not catch 429
if (response.ok) {
console.log('ok');
return response.json();
} else {
console.log(`FAILED: ${link}`);
// Return a Promise
return Promise.reject(`Error when getting venue details for '${venueId}'`);
}
});
});
return Promise.all(venuePromises);
})
.then(venueData => {
const venues = venueData.map(entry => entry.response.venue);
this.parseFsqData(venues);
})
.catch(e => {console.log(e); getBackupData()});
function getBackupData() {
console.log('backup')
}
Try returning the rejected promise.
return Promise.reject('Error when getting venue details')
When working with promises you should return inner promises instad of working with inner "thens".
Check this:
fetch(`https://api.foursquare.com/v2/venues/search?${params}`)
.then(response => response.json())
.then(data => {
const venueIds = data.response.venues.map(venue => venue.id);
const venuePromises = venueIds.map(venueId => {
fetch(`https://api.foursquare.com/v2/venues/${venueId}?${otherParams}`)
.then(response => {
// Must check for response.ok, because
// catch() does not catch 429
if (response.ok) {
console.log('ok')
return response.json()
} else {
return Promise.reject('Error when getting venue details')
}
})
});
return Promise.all(venuePromises)
})
.then(venueValues => {
const venues = venueValues.map(entry => entry.response.venue); // Error for this line
this.parseFsqData(venues);
})
.catch((e) => {console.log(e); getBackupData()})
function getBackupData() {
console.log('backup')
}
When returning Promise.all as a value, you're returning a promise so that you can chain further "then" callbacks. The last catch shall capture all promise rejects.
You're also missing the return in the else clause
Hope this helps
I believe the solution is fairly simple; The response of the nested fetch method is missing a return statement. You should get rid of that mysterious error once it is in place.
const venuePromises = venueIds.map(venueId => {
<missing return statement here> fetch(`https://api.foursquare.com/v2/venues/${venueId}?${otherParams}`)
.then(response => {
tl;dr - if you have to filter the promises (say for errored ones) don't use async functions
I'm trying to fetch a list of urls with async and parse them, the problem is that if there's an error with one of the urls when I'm fetching - let's say for some reason the api endpoint doesn't exists - the program crushes on the parsing with the obvious error:
UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): TypeError: ext is not iterable
I've tried checking if the res.json() is undefined, but obviously that's not it as it complains about the entire 'ext' array of promises not being iterable.
async function fetchAll() {
let data
let ext
try {
data = await Promise.all(urls.map(url=>fetch(url)))
} catch (err) {
console.log(err)
}
try {
ext = await Promise.all(data.map(res => {
if (res.json()==! 'undefined') { return res.json()}
}))
} catch (err) {
console.log(err)
}
for (let item of ext) {
console.log(ext)
}
}
Question 1:
How do I fix the above so it won't crash on an invalid address?
Question 2:
My next step is to write the extracted data to the database.
Assuming the data size of 2-5mgb of content, is my approach of using Promise.all() memory efficient? Or will it be more memory efficient and otherwise to write a for loop which handles each fetch then on the same iteration writes to the database and only then handles the next fetch?
You have several problems with your code on a fundamental basis. We should address those in order and the first is that you're not passing in any URLS!
async function fetchAll(urls) {
let data
let ext
try {
data = await Promise.all(urls.map(url=>fetch(url)))
} catch (err) {
console.log(err)
}
try {
ext = await Promise.all(data.map(res => {
if (res.json()==! 'undefined') { return res.json()}
}))
} catch (err) {
console.log(err)
}
for (let item of ext) {
console.log(ext)
}
}
First you have several try catch blocks on DEPENDANT DATA. They should all be in a single try catch block:
async function fetchAll(urls) {
try {
let data = await Promise.all(urls.map(url=>fetch(url)))
let ext = await Promise.all(data.map(res => {
// also fixed the ==! 'undefined'
if (res.json() !== undefined) { return res.json()}
}))
for (let item of ext) {
console.log(ext)
}
} catch (err) {
console.log(err)
}
}
Next is the problem that res.json() returns a promise wrapped around an object if it exists
if (res.json() !== undefined) { return res.json()}
This is not how you should be using the .json() method. It will fail if there is no parsable json. You should be putting a .catch on it
async function fetchAll(urls) {
try {
let data = await Promise.all(urls.map(url => fetch(url).catch(err => err)))
let ext = await Promise.all(data.map(res => res.json ? res.json().catch(err => err) : res))
for (let item of ext) {
console.log(ext)
}
} catch (err) {
console.log(err)
}
}
Now when it cannot fetch a URL, or parse a JSON you'll get the error and it will cascade down without throwing. Now your try catch block will ONLY throw if there is a different error that happens.
Of course this means we're putting an error handler on each promise and cascading the error, but that's not exactly a bad thing as it allows ALL of the fetches to happen and for you to distinguish which fetches failed. Which is a lot better than just having a generic handler for all fetches and not knowing which one failed.
But now we have it in a form where we can see that there is some better optimizations that can be performed to the code
async function fetchAll(urls) {
try {
let ext = await Promise.all(
urls.map(url => fetch(url)
.then(r => r.json())
.catch(error => ({ error, url }))
)
)
for (let item of ext) {
console.log(ext)
}
} catch (err) {
console.log(err)
}
}
Now with a much smaller footprint, better error handling, and readable, maintainable code, we can decide what we eventually want to return. Now the function can live wherever, be reused, and all it takes is a single array of simple GET URLs.
Next step is to do something with them so we probably want to return the array, which will be wrapped in a promise, and realistically we want the error to bubble since we've handled each fetch error, so we should also remove the try catch. At that point making it async no longer helps, and actively harms. Eventually we get a small function that groups all URL resolutions, or errors with their respective URL that we can easily filter over, map over, and chain!
function fetchAll(urls) {
return Promise.all(
urls.map(url => fetch(url)
.then(r => r.json())
.then(data => ({ data, url }))
.catch(error => ({ error, url }))
)
)
}
Now we get back an array of similar objects, each with the url it fetched, and either data or an error field! This makes chaining and inspecting SUPER easy.
You are getting a TypeError: ext is not iterable - because ext is still undefined when you caught an error and did not assign an array to it. Trying to loop over it will then throw an exception that you do not catch.
I guess you're looking for
async function fetchAll() {
try {
const data = await Promise.all(urls.map(url => fetch(url)));
const ext = await Promise.all(data.map(res => res.json()));
for (let item of ext) {
console.log(item);
}
} catch (err) {
console.log(err);
}
}
Instead of fetch(url) on line 5, make your own function, customFetch, which calls fetch but maybe returns null, or an error object, instead of throwing.
something like
async customFetch(url) {
try {
let result = await fetch(url);
if (result.json) return await result.json();
}
catch(e) {return e}
}
if (res.json()==! 'undefined')
Makes no sense whatsoever and is an asynchronous function. Remove that condition and just return res.json():
try {
ext = await Promise.all(data.map(res => res.json()))
} catch (err) {
console.log(err)
}
Whether or not your approach is "best" or "memory efficient" is up for debate. Ask another question for that.
You can have fetch and json not fail by catching the error and return a special Fail object that you will filter out later:
function Fail(reason){this.reason=reason;};
const isFail = o => (o&&o.constructor)===Fail;
const isNotFail = o => !isFail(o);
const fetchAll = () =>
Promise.all(
urls.map(
url=>
fetch(url)
.then(response=>response.json())
.catch(error=>new Fail([url,error]))
)
);
//how to use:
fetchAll()
.then(
results=>{
const successes = results.filter(isNotFail);
const fails = results.filter(isFail);
fails.forEach(
e=>console.log(`failed url:${e.reason[0]}, error:`,e.reason[1])
)
}
)
As for question 2:
Depending on how many urls you got you may want to throttle your requests and if the urls come from a large file (gigabytes) you can use stream combined with the throttle.
async function fetchAll(url) {
return Promise.all(
url.map(
async (n) => fetch(n).then(r => r.json())
)
);
}
fetchAll([...])
.then(d => console.log(d))
.catch(e => console.error(e));
Will this work for you?
If you don't depend on every resource being a success I would have gone back to basics skipping async/await
I would process each fetch individual so I could catch the error for just the one that fails
function fetchAll() {
const result = []
const que = urls.map(url =>
fetch(url)
.then(res => res.json())
.then(item => {
result.push(item)
})
.catch(err => {
// could't fetch resource or the
// response was not a json response
})
)
return Promise.all(que).then(() => result)
}
Something good #TKoL said:
Promise.all errors whenever one of the internal promises errors, so whatever advice anyone gives you here, it will boil down to -- Make sure that you wrap the promises in an error handler before passing them to Promise.all
Regarding question 1, please refer to this:
Handling errors in Promise.all
Promise.all is all or nothing. It resolves once all promises in the array resolve, or reject as soon as one of them rejects. In other words, it either resolves with an array of all resolved values, or rejects with a single error.
Is this the only way to use the body.json() and also get the status code?
let status;
return fetch(url)
.then((response => {
status = response.status;
return response.json()
})
.then(response => {
return {
response: response,
status: status
}
});
This doesn't work as it returns a promise in the response field:
.then((response)=> {return {response: response.json(), status: response.status}})
Your status is not visible in the second then. You can just get the two properties in the single then.
json() returns a new Promise to you, so you need to create your object inside the then of the result of that function. If you return a Promise from a function, it will be fulfilled and will return the result of the fulfillment - in our case the object.
fetch("https://jsonplaceholder.typicode.com/posts/1")
.then(r => r.json().then(data => ({status: r.status, body: data})))
.then(obj => console.log(obj));
The .json method returns a promise, not the parsed value itself. If you want to access both the response and the parsed value in the same callback, you'll need to use nested functions like this:
fetch(url)
.then(response => {
response.json().then(parsedValue => {
// code that can access both here
})
});
Alternatively, you can use await inside an asynchronous function to eliminate the need for callbacks.
const response = await fetch(url);
const parsedValue = await response.json();
// code that can access both here
Of course, you'll want to check for errors, either with a .catch(...) call on a Promise or with a try...catch block in an async function. You could make a function that handles JSON and error cases, and then reuse it for all fetches. For example, something like this:
function handle(response) {
if (response.ok) {
return response.json().then(parsedValue => {
// the status was ok and the body could be parsed
return Promise.resolve({ response, parsedValue });
}).catch(err => {
// the status was ok but the body was empty or not JSON
return Promise.resolve({ response });
});
} else {
return response.json().catch(err => {
// the status was not ok and the body was unobtainable/empty/not JSON
throw new Error(response.statusText);
}).then(parsedValue => {
// the status was not ok and the body was JSON
throw new Error(parsedValue.error.message); // assuming an error message is returned by our REST API
});
}
}
I don't think it's the best design pattern, but hopefully this clarifies how the fetch API works.
PS: I avoided naming any variable or property json since that is the name of the text format. Once it's been parsed, it is no longer JSON.
Using two 'then's seem unnecessary to me.
async/await could get the job done pretty easily.
fetch('http://test.com/getData')
.then( async (response) => {
// get json response here
let data = await response.json();
if(data.status === 200){
// Process data here
}else{
// Rest of status codes (400,500,303), can be handled here appropriately
}
})
.catch((err) => {
console.log(err);
})
Did you try this?
return fetch(url)
.then((r)=> {return {response: r.json(), status: r.status}})
I think the cleanest way is to create a Promise.all() with the pieces you need.
.then(response => Promise.all([Promise.resolve(response.ok), response.text()]))
Which can be written shorter as
.then(response => Promise.all([response.ok, response.text()]))
The promise returns an array with all of the results
.then(data => ({ status: data[0], response: data[1] }))