I'm quite a newbie in JavaScript and in Promises.
I'm trying to build an array of objects that I get from an API.
To do so, I've build two functions in a file MyFile.js.
The first one returns a promise when an axios promise is resolved. It's
function get_items (url) {
return new Promise((resolve, reject) => {
let options = {
baseURL: url,
method: 'get'
}
axios(options)
.then(response => {
resolve(response.data)
})
.catch(error => {
reject(error.stack)
})
})
}
The second one looks like this:
let output = []
let next_url = 'https://some_url.com/api/data'
async function get_data () {
try {
let promise = new Promise((resolve, reject) => {
if (next_url) {
get_items(next_url)
.then(response => {
output.push(...response.results)
if (response.next) {
next_url = response.next
console.log('NEXT_URL HERE', next_url)
get_data()
} else {
console.log('else')
next_url = false
get_data()
}
})
.catch(error => {
reject(error.stack)
})
} else {
console.log('before resolve')
resolve(output)
}
})
return await promise
} catch(e) {
console.log(e)
}
}
It's where I'm grinding my teeth.
What I think I understand of this function, is that:
it's returning the value of a promise (that's what I understand return await promise is doing)
it's a recursive function. So, if there is a next_url, the function continues on. But if there is not, it gets called one last time to go into the else part where it resolves the array output which contains the results (values not state) of all the promises. At least, when I execute it, and check for my sanity checks with the console.log I wrote, it works.
So, output is filled with data and that's great.
But, when I call this function from another file MyOtherFile.js, like this:
final_output = []
MyFile.get_data()
.then(result => {
console.log('getting data')
final_output.push(...result)
})
it never gets into the then part. And when I console.log MyFile.get_data(), it's a pending promise.
So, what I would like to do, is be able to make get_data() wait for all the promises result (without using Promise.all(), to have calls in serie, not in parallel, that would be great for performances, I guess?) and then be able to retrieve that response in the then part when calling this function from anywhere else.
Keep in mind that I'm really a newbie in promises and JavaScript in general (I'm more of a Python guy).
Let me know if my question isn't clear enough.
I've been scratching my head for two days now and it feels like I'm running in circle.
Thanks for being an awesome community!
This is a bit untested
const api_url = 'https://some_url.com/api/data';
get_data(api_url).then((results) => {
console.log(results);
}).catch((error) => {
// console.error(error);
});
function get_items (url) {
const options = {
baseURL: url,
method: 'get'
};
return axios(options).then((response) => response.data);
}
async function get_data(next_url) {
const output = [];
while (next_url) {
const { results, next } = await get_items(next_url);
output.push(...results);
next_url = next;
}
return output;
}
Basically it makes things a bit neater. I suggest to look at more examples with Promises and the advantage and when to ease await/async. One thing to keep in mind, if you return a Promise, it will follow the entire then chain, and it will always return a Promise with a value of the last then.. if that makes sense :)
There are a few problems. One is that you never resolve the initial Promise unless the else block is entered. Another is that you should return the recursive get_data call every time, so that it can be properly chained with the initial Promise. You may also consider avoiding the explicit promise construction antipattern - get_items already returns a Promise, so there's no need to construct another one (same for the inside of get_items, axios calls return Promises too).
You might consider a plain while loop, reassigning the next_url string until it's falsey:
function get_items (baseURL) {
const options = {
baseURL: url,
method: 'get'
}
// return the axios call, handle errors in the consumer instead:
return axios(options)
.then(res => res.data)
}
async function get_data() {
const output = []
let next_url = 'https://some_url.com/api/data'
try {
while (next_url) {
const response = await get_items(next_url);
output.push(...response.results)
next_url = response.next;
}
} catch (e) {
// handle errors *here*, perhaps
console.log(e)
}
return output;
}
Note that .catch will result in a Promise being converted from a rejected Promise to a resolved one - you don't want to .catch everywhere, because that will make it difficult for the caller to detect errors.
Another way of doing it is to not use async at all and just recursively return a promise:
const getItems = (url) =>
axios({
baseURL: url,
method: 'get',
}).then((response) => response.data);
const getData = (initialUrl) => {
const recur = (result, nextUrl) =>
!nextUrl
? Promise.resolve(result)
: getItems(nextUrl).then((data) =>
recur(result.concat([data.results]), data.next),
);
return recur([],initialUrl)
.catch(e=>Promise.reject(e.stack));//reject with error stack
};
As CertainPerformance noted; you don't need to catch at every level, if you want getData to reject with error.stack you only need to catch it once.
However; if you had 100 next urls and 99 of them were fine but only the last one failed would you like to reject in a way that keeps the results so far so you can try again?
If you do then the code could look something like this:
const getData = (initialUrl) => {
const recur = (result, nextUrl) =>
!nextUrl
? Promise.resolve(result)
: getItems(nextUrl)
.catch(e=>Promise.reject([e,result]))//reject with error and result so far
.then((data) =>
recur(result.concat([data.results]), data.next),
);
return recur([],initialUrl);//do not catch here, just let it reject with error and result
};
Related
Desperately trying to write a sync version of https://www.npmjs.com/package/node-firebird#reading-blobs-aasynchronous
Basically I need to (a)wait twice:
for the callback function to execute so that the eventEmitter is available
for the "end" event to occur
and then return the Buffer.
my code (JS/TS mix for now) currently does 2, but not 1 : readBlob returns undefined, then Buffer.concat(buffers) is called later ... :
function readBLOB(callback: any): Buffer {
return callback(async (err, _, eventEmitter) => {
let buffers = []
if (err)
throw err
eventEmitter.on('data', chunk => {
buffers.push(chunk);
});
return await eventEmitter.once('end', function (e) {
return Buffer.concat(buffers)
})
})
}
Sorry to ask one more time (yes, I checked a lot of other questions and tried a lot of things...), but how to make this work (simply...) ?
(the function that calls the callback is fetch_blob_async in https://github.com/hgourvest/node-firebird/blob/master/lib/index.js#L4261 , just in case...)
There are few mistakes here like returning an callback function, witch returns, i guess, undefined or returning something IN an callback function that makes no sense.
Also async / await makes no sense here it has no effect. async / await is only useful if you want to await till some Promise resolves. But you have no Promise in your code at all.
What you need is new Promise
function readBLOB(callback) {
return new Promise((resolve, reject) => {
callback((err, _, eventEmitter) => {
let buffers = [];
if (err) reject(err);
eventEmitter.on("data", chunk => {
buffers.push(chunk);
});
eventEmitter.once("end", function(e) {
resolve(Buffer.concat(buffers));
});
});
});
}
Simple like that. You resolve your Buffer and reject if some error occurs
Now you can use it like:
readBLOB(cb).then(data => {
console.log(data);
})
At one point, we are calling our backend several times with fetch. Each call returns a promise, which we collect and pass to our responseHandler. This handler is supposed to dispatch redux actions when each response comes in. The action will be shaped differently based on two factors:
the response code of the server
the content of the request
so we have this
function save(objects) {
let promises = objects.map(object => {
let promise = fetch(url, {..., body: object});
});
responseHandler(promises);
}
function responseHandler(promises) {
for (let promise of promises) {
promise
.then(r => {
return r.json();
})
.then(body => dispatchSomeAction(body));
}
}
So far so good. But now we need info from inside of the save and from inside of the anonymous function that receives the response object. So I came up with the following approach. But it feels wrong to piggyback on the promise object. I can't say why.
function save(objects) {
let promises = objects.map(object => {
let promise = fetch(url, {..., body: object});
promise._objectType = object._type;
});
responseHandler(promises);
}
function responseHandler(promises) {
for (let promise of promises) {
promise
.then(r => {
promise._status = r.status;
return r.json();
})
.then(body => dispatchSomeAction({
status: promise._status,
type: promise._type, ...body
}));
}
}
As can be seen, every time I want a value of a scope to be available later, I place it on the promise object because that promise object is in all the other scopes. Is there a cleaner way?
Note: The backend returns an object that doesn't hold the _type property (actually I clear my objects of those flags before sending them to the backend. It's a frontend flag that I still require when the answer comes)
You must carry the object. Using ES6 structuring and destructuring is not very clumpsy:
function save(objects){
let promises = objects.map(object => {
return fetch(url, {..., body: object}).then(r => ({r,type:object._type}));
}
responseHandler(promises)
}
function responseHandler(promises){
for (let promise of promises){
promise
.then(({r,type}) => {
return r.json().then(body => dispatchSomeAction({status: r.status, type, ...body));
});
}
}
Note that you don't really need to do anything at all with the status, as r it was still in scope where it's used.
I would try something like the following
function save(objects){
let promises = objects.map(object) =>{
return fetch(url, {..., body: object}).then(r=>{
return {data: r, originalObject: object}
})
}
responseHandler(promises)
}
function responseHandler(promises){
for (let promise of promises){
promise
.then(r => {
return {data: r.data.json(), originalObject: r.originalObject)
.then(r=> dispatchSomeAction({status: r.data.status, type: r.originalObject.type, ...r.data.body));
}
}
you can tweak a little bit, say, without passing the whole object, just the type field.
A common pattern is to use a single global object (commonly called state or appData or similar) to store values and allow you to access them from within different closures.
Here's a contrived example of using this pattern to load a bunch of items, and then to reload the data for each item after clicking a "refresh" button:
const state = {}
getItemsPromise().then(response => {
state.items = response.items
})
$(refreshItemsButton).on('click', ()=>{
state.items.forEach(item => {
refreshItemPromise(item).then(response => {
state.items[item.id] == response.item
})
})
})
I've been trying to figure out what the proper way would be to write a promise for this function. I have an asynchronous function that makes an HTTP request to the server to retrieve a response, "documents_fileUploader." I am mapping through the "url" of each item within the response, and each url will go in to a function that makes another HTTP request and then sets the state. I want to fire the "upload()" function only after everything within the "documents_fileUploader()" function is complete. I tried doing this without a promise and it went straight to my "upload()" function because request was still pending. Any suggestions on how to go about this?
documents_fileUploader(formData).then(resp => {
resp.data.items.map(url => {
const key = url.split("/")[4];
this.setState({
urls: [...this.state.urls, url],
keys: [...this.state.keys, key]
});
this.getFileObject(key);
})
}).then(() => {
this.upload();
})
getFileObject = file => {
file_view(file).then(resp => {
this.setState({
mimeTypes: [...this.state.mimeTypes, resp.data.item.headers.contentType]
})
}).catch(err => {
console.log(err);
})
}
To your main question, you can wait for every promise that your .map call returns by using the Promise.all method.
Second, in order for that to work, your getFileObject function must return the promise it creates.
So incorporating those two changes, your snippet might look like:
documents_fileUploader(formData).then(resp => {
return Promise.all(resp.data.items.map(url => { // Wrap in Promise.all and return it
const key = url.split("/")[4];
this.setState({
urls: [...this.state.urls, url],
keys: [...this.state.keys, key]
});
return this.getFileObject(key); // Make sure to return this promise as well.
}));
}).then(() => {
// Now this won't happen until every `getFileObject` promise has resolved...
this.upload();
})
getFileObject = file => {
return file_view(file).then(resp => { // And return the promise here.
this.setState({
mimeTypes: [...this.state.mimeTypes, resp.data.item.headers.contentType]
})
}).catch(err => {
console.log(err);
})
}
tl;dr - if you have to filter the promises (say for errored ones) don't use async functions
I'm trying to fetch a list of urls with async and parse them, the problem is that if there's an error with one of the urls when I'm fetching - let's say for some reason the api endpoint doesn't exists - the program crushes on the parsing with the obvious error:
UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): TypeError: ext is not iterable
I've tried checking if the res.json() is undefined, but obviously that's not it as it complains about the entire 'ext' array of promises not being iterable.
async function fetchAll() {
let data
let ext
try {
data = await Promise.all(urls.map(url=>fetch(url)))
} catch (err) {
console.log(err)
}
try {
ext = await Promise.all(data.map(res => {
if (res.json()==! 'undefined') { return res.json()}
}))
} catch (err) {
console.log(err)
}
for (let item of ext) {
console.log(ext)
}
}
Question 1:
How do I fix the above so it won't crash on an invalid address?
Question 2:
My next step is to write the extracted data to the database.
Assuming the data size of 2-5mgb of content, is my approach of using Promise.all() memory efficient? Or will it be more memory efficient and otherwise to write a for loop which handles each fetch then on the same iteration writes to the database and only then handles the next fetch?
You have several problems with your code on a fundamental basis. We should address those in order and the first is that you're not passing in any URLS!
async function fetchAll(urls) {
let data
let ext
try {
data = await Promise.all(urls.map(url=>fetch(url)))
} catch (err) {
console.log(err)
}
try {
ext = await Promise.all(data.map(res => {
if (res.json()==! 'undefined') { return res.json()}
}))
} catch (err) {
console.log(err)
}
for (let item of ext) {
console.log(ext)
}
}
First you have several try catch blocks on DEPENDANT DATA. They should all be in a single try catch block:
async function fetchAll(urls) {
try {
let data = await Promise.all(urls.map(url=>fetch(url)))
let ext = await Promise.all(data.map(res => {
// also fixed the ==! 'undefined'
if (res.json() !== undefined) { return res.json()}
}))
for (let item of ext) {
console.log(ext)
}
} catch (err) {
console.log(err)
}
}
Next is the problem that res.json() returns a promise wrapped around an object if it exists
if (res.json() !== undefined) { return res.json()}
This is not how you should be using the .json() method. It will fail if there is no parsable json. You should be putting a .catch on it
async function fetchAll(urls) {
try {
let data = await Promise.all(urls.map(url => fetch(url).catch(err => err)))
let ext = await Promise.all(data.map(res => res.json ? res.json().catch(err => err) : res))
for (let item of ext) {
console.log(ext)
}
} catch (err) {
console.log(err)
}
}
Now when it cannot fetch a URL, or parse a JSON you'll get the error and it will cascade down without throwing. Now your try catch block will ONLY throw if there is a different error that happens.
Of course this means we're putting an error handler on each promise and cascading the error, but that's not exactly a bad thing as it allows ALL of the fetches to happen and for you to distinguish which fetches failed. Which is a lot better than just having a generic handler for all fetches and not knowing which one failed.
But now we have it in a form where we can see that there is some better optimizations that can be performed to the code
async function fetchAll(urls) {
try {
let ext = await Promise.all(
urls.map(url => fetch(url)
.then(r => r.json())
.catch(error => ({ error, url }))
)
)
for (let item of ext) {
console.log(ext)
}
} catch (err) {
console.log(err)
}
}
Now with a much smaller footprint, better error handling, and readable, maintainable code, we can decide what we eventually want to return. Now the function can live wherever, be reused, and all it takes is a single array of simple GET URLs.
Next step is to do something with them so we probably want to return the array, which will be wrapped in a promise, and realistically we want the error to bubble since we've handled each fetch error, so we should also remove the try catch. At that point making it async no longer helps, and actively harms. Eventually we get a small function that groups all URL resolutions, or errors with their respective URL that we can easily filter over, map over, and chain!
function fetchAll(urls) {
return Promise.all(
urls.map(url => fetch(url)
.then(r => r.json())
.then(data => ({ data, url }))
.catch(error => ({ error, url }))
)
)
}
Now we get back an array of similar objects, each with the url it fetched, and either data or an error field! This makes chaining and inspecting SUPER easy.
You are getting a TypeError: ext is not iterable - because ext is still undefined when you caught an error and did not assign an array to it. Trying to loop over it will then throw an exception that you do not catch.
I guess you're looking for
async function fetchAll() {
try {
const data = await Promise.all(urls.map(url => fetch(url)));
const ext = await Promise.all(data.map(res => res.json()));
for (let item of ext) {
console.log(item);
}
} catch (err) {
console.log(err);
}
}
Instead of fetch(url) on line 5, make your own function, customFetch, which calls fetch but maybe returns null, or an error object, instead of throwing.
something like
async customFetch(url) {
try {
let result = await fetch(url);
if (result.json) return await result.json();
}
catch(e) {return e}
}
if (res.json()==! 'undefined')
Makes no sense whatsoever and is an asynchronous function. Remove that condition and just return res.json():
try {
ext = await Promise.all(data.map(res => res.json()))
} catch (err) {
console.log(err)
}
Whether or not your approach is "best" or "memory efficient" is up for debate. Ask another question for that.
You can have fetch and json not fail by catching the error and return a special Fail object that you will filter out later:
function Fail(reason){this.reason=reason;};
const isFail = o => (o&&o.constructor)===Fail;
const isNotFail = o => !isFail(o);
const fetchAll = () =>
Promise.all(
urls.map(
url=>
fetch(url)
.then(response=>response.json())
.catch(error=>new Fail([url,error]))
)
);
//how to use:
fetchAll()
.then(
results=>{
const successes = results.filter(isNotFail);
const fails = results.filter(isFail);
fails.forEach(
e=>console.log(`failed url:${e.reason[0]}, error:`,e.reason[1])
)
}
)
As for question 2:
Depending on how many urls you got you may want to throttle your requests and if the urls come from a large file (gigabytes) you can use stream combined with the throttle.
async function fetchAll(url) {
return Promise.all(
url.map(
async (n) => fetch(n).then(r => r.json())
)
);
}
fetchAll([...])
.then(d => console.log(d))
.catch(e => console.error(e));
Will this work for you?
If you don't depend on every resource being a success I would have gone back to basics skipping async/await
I would process each fetch individual so I could catch the error for just the one that fails
function fetchAll() {
const result = []
const que = urls.map(url =>
fetch(url)
.then(res => res.json())
.then(item => {
result.push(item)
})
.catch(err => {
// could't fetch resource or the
// response was not a json response
})
)
return Promise.all(que).then(() => result)
}
Something good #TKoL said:
Promise.all errors whenever one of the internal promises errors, so whatever advice anyone gives you here, it will boil down to -- Make sure that you wrap the promises in an error handler before passing them to Promise.all
Regarding question 1, please refer to this:
Handling errors in Promise.all
Promise.all is all or nothing. It resolves once all promises in the array resolve, or reject as soon as one of them rejects. In other words, it either resolves with an array of all resolved values, or rejects with a single error.
Let's say I have two async events, both need to i/o with remote exchange.
placeOrder()
cancelOrder()
Both events fire in async way, which means cancelOrder can be called before placeOrder return. Tricky part is I need the placeOrder to return an Order ID first otherwise there is no way to call cancelOrder, so I need some way to block the cancelOrder event right until placeOrder returns, and the blockage cannot be too long otherwise the Order may be executed, so loop/timeout/frequent checking doesn't work here.
Any idea?
You would use a Promise for that. If your functions already return a promise, you can simply chain the both functions using then()
placeOrder().then(val => cancelOrder(val));
If they do not, you can put them inside a new Promise
function foo() {
return new Promise((resolve, reject) => {
// do stuff
resolve('<result of placeOrder here>');
});
}
function bar(val) {
return new Promise((resolve, reject) => {
// do stuff
resolve('whatever')
})
}
and call
foo()
.then(value => bar(value))
.then(console.log);
If you are able to use ES2017, the you can use async functions. For example, I'm going to assume that your functions perform some sort of request to the database using fetch or axios since you haven't specified. Then you can write placeOrder and cancelOrder like so:
const placeOrder = async () => {
try {
const response = await fetch('/place_order');
// Do something with the response
} catch (err) {
// Handle error
}
};
const cancelOrder = async () => {
try {
const response = await fetch('/cancel_order');
// Do something with the response
} catch (err) {
// Handle error
}
};
const someFunction = async () => {
await placeOrder();
await cancelOrder();
};