Recursive JavaScript Promises (auto paging an API) - javascript

I have a paged API that I need to automatically fetch each page of results.
I have constructed the following recursive promise chain, which actually gives me the desired output. I have renamed the traditional resolve to outer to try and wrap my head around it a little more but I am still confused at how passing the outer function into the nested .then is the magic sauce that is allowing this to work?
I thought I knew Promises well but obviously not!
protected execute<T>(httpOptions): Promise<AxiosResponse | T> {
const reqObject: HttpConfig = {
...httpOptions,
baseURL: this.config.baseUri,
headers: { Authorization: `Bearer ${this.config.apiKey}` },
};
return new Promise((outer, reject) => {
axios
.request<T>(reqObject)
.then((res) => {
if (res.headers.link) {
const pagingObject: PagingObject = this.getPagingObject(res);
this.pagedListResponses = this.pagedListResponses.concat(res.data);
if (pagingObject.last) {
this.execute<T>({
url: pagingObject.next,
}).then(outer);
} else {
outer(this.pagedListResponses as any);
}
} else {
outer("No paging header found");
}
})
.catch((err) => reject(err));
});
}

TL:DR
As you add more and more Promises via recursion they pile up like the skin of an onion. When each is resolved and .then() is called, the Promises are removed from the inside out until there are none left.
Promise Chaining
When a Promise object is created within the Resolve handler
of another Promise both Promises are settled in an inside-out order. Additionally, when chaining is used through
the Promise instance methods (.then(), .catch(), .always(), etc), the chaining methods take priority and are
executed before the outermost Promise object resolves.
Perhaps better explained here.
Your code creates the Axios Promise within the Resolve handler of the outer construced Promise Object. The
AxiosPromise's .then() will execute prior to the outer Promise finally settling. After that happens the result is passed through the outer Promise object with no modification or processing. Basically a no-op.
That is why the wrapper Promise (explicit-construction anti-pattern)
is unecessary and discouraged - it is a waste of time and resources that provides no benefit in this code.
With recursion in the mix, Promise objects just keep getting piled on.
So (for now) a reference to that outer Promise object is returned from the .execute() method. But when/how is it settled (resolved or rejected)?
The Axios Promise resolves with AJAX results
If this is a recursive call, this is an inner Promise (even though created with .then(outer) which is more confusing in this case)
The outer most .then() is called with the AJAX results (or rejected with reason)
The instance variable pagedListResponses is updated with the results
if res.headers.link == true && pageObject.last == true
Recurse & start back at 1 <- pending with unsettled Promise(s)
else if res.headers.link == true && pageObject.last == false
resolve the outer most Promise with the pageListResponses <- completely settled
else if res.headers.link == false && pageObject.last == false
resolve the outer most Promise with "No paging header found" <- completely settled
Then then() method attached to the initial call to execute() is called with the pageListResponses
ex. this.execute({...}).then(pageList=>doSomethingWithPageOfResults());
So chaining, using .then() midstream, allows us to do some data processing (as you have done) prior to returning an
eventually settled Promise result.
In the recursive code:
this.execute<T>({
url: pagingObject.next,
}).then(outer);
The .then() call here simply adds a new inner Promise to the chain and, as you realize, is exactly the same as writing:
.then(result=>outer(result)); Reference
Async/Await
Finally, using Async / Await is recommended. It is strongly suggested to Rewrite Promise code with Async/Await for complex (or some say any) scenarios.
Although still asychronous, this makes reading and rationalizing about the code sequence much easier.
Your code rewritten to take advantage of Async/Await:
protected async execute<T>(httpOptions): Promise<T> {
const reqObject = {
...httpOptions,
baseURL: this.config.baseUri,
headers: {Authorization: `Bearer ${this.config.apiKey}`},
};
try {
const res: AxiosResponse<T> = await axios.request<T>(reqObject);
if (res) {
const pagingObject: boolean = this.getPagingObject(res);
this.pagedListResponses = this.pagedListResponses.concat(res.data);
if (pagingObject) {
return this.execute<T>({ url: 'https://jsonplaceholder.typicode.com/users/1/todos'});
} else {
return this.pagedListResponses as any;
}
} else {
return "No paging header found" as any;
}
} catch (err) {
throw new Error('Unable to complete request for data.');
}
}
Here, async directly indicates "This method returns a Promise" (as typed). The await keyword is used in code only once to wait for the resolution
of the AJAX request. That statement returns the actual result (as typed) as opposed to a Promise object. What happens with a network error? Note the use of try/catch here. The catch block handles any reject in the nest/chain. You might notice the recursive call has no await keyword. It isn't necessary in this case since it is a Promise that just passes along its result. (it's OK to add await here but it has no real benefit).
This code is used the same way from the outside:
this.execute({...}).then(pageList=>doSomethingWithPageOfResults());
Let me know what gaps you may still have.

Well when you call this.execute, it returns a Promise<AxiosResponse | T>.
So by calling .then(outer), the response from the recursive execute function is being used to resolve your outer promise.
It's basically the same as this.execute(...).then(response => outer(response)), if that makes it any clearer.

Related

Object Promise, impossible to get value

I have some asynchronous function:
async global(){
return 'Test success'
}
I call this in my hook and it must be fulfilled:
const Controller = use('App/Controllers/Http/Controller')
View.global('ShowGlobal', async () => {
const call = new Controller()
const info = await call.global()
console.log(info)
return info
})
At the same time i am getting correct console.log with result 'Test success', but return gives me constantly [object Promise]
What am I doing wrong?
(This is an answer I found)
The code you write in JavaScript is run on one thread, that means that if your code could actually wait for something it will block any of your other code from getting executed. The event loop of JavaScript is explained very well in this video and if you like to read in this page.
A good example of blocking code in the browser is alert("cannot do anything until you click ok");. Alert blocks everything, the user can't even scroll or click on anything in the page and your code also blocks from executing.
Promise.resolve(22)
.then(x=>alert("blocking")||"Hello World")
.then(
x=>console.log(
"does not resolve untill you click ok on the alert:",
x
)
);
Run that in a console and you see what I mean by blocking.
This creates a problem when you want to do something that takes time. In other frameworks you'd use a thread or processes but there is no such thing in JavaScript (technically there is with web worker and fork in node but that's another story and usually far more complicated than using async api's).
So when you want to make a http request you can use fetch but fetch takes some time to finish and your function should not block (has to return something as fast as possible). This is why fetch returns a promise.
Note that fetch is implemented by browser/node and does run in another thread, only code you write runs in one thread so starting a lot of promises that only run code you write will not speed up anything but calling native async api's in parallel will.
Before promises async code used callbacks or would return an observable object (like XmlHttpRequest) but let's cover promises since you can convert the more traditional code to a promise anyway.
A promise is an object that has a then function (and a bunch of stuff that is sugar for then but does the same), this function takes 2 parameters.
Resolve handler: A function that will be called by the promise when the promise resolves (has no errors and is finished). The function will be passed one argument with the resolve value (for http requests this usually is the response).
Reject handler: A function that will be called by the promise when the promise rejects (has an error). This function will be passed one argument, this is usually the error or reason for rejection (can be a string, number or anything).
Converting callback to promise.
The traditional api's (especially nodejs api's) use callbacks:
traditionalApi(
arg
,function callback(err,value){
err ? handleFail(err) : processValue(value);
}
);
This makes it difficult for the programmer to catch errors or handle the return value in a linear way (from top to bottom). It gets even more impossible to try and do things parallel or throttled parallel with error handling (impossible to read).
You can convert traditional api's to promises with new Promise
const apiAsPromise = arg =>
new Promise(
(resolve,reject)=>
traditionalApi(
arg,
(err,val) => (err) ? reject(err) : resolve(val)
)
)
async await
This is what's called syntax sugar for promises. It makes promise consuming functions look more traditional and easier to read. That is if you like to write traditional code, I would argue that composing small functions is much easier to read. For example, can you guess what this does?:
const handleSearch = search =>
compose([
showLoading,
makeSearchRequest,
processRespose,
hideLoading
])(search)
.then(
undefined,//don't care about the resolve
compose([
showError,
hideLoading
])
);
Anayway; enough ranting. The important part is to understand that async await doesn't actually start another thread, async functions always return a promise and await doesn't actually block or wait. It's syntax sugar for someFn().then(result=>...,error=>...) and looks like:
async someMethod = () =>
//syntax sugar for:
//return someFn().then(result=>...,error=>...)
try{
const result = await someFn();
...
}catch(error){
...
}
}
The examples allways show try catch but you don't need to do that, for example:
var alwaysReject = async () => { throw "Always returns rejected promise"; };
alwaysReject()
.then(
x=>console.log("never happens, doesn't resolve")
,err=>console.warn("got rejected:",err)
);
Any error thrown or await returning a rejected promise will cause the async function to return a rejected promise (unless you try and catch it). Many times it is desirable to just let it fail and have the caller handle errors.
Catching errors could be needed when you want the promise to succeed with a special value for rejected promises so you can handle it later but the promise does not technically reject so will always resolve.
An example is Promise.all, this takes an array of promises and returns a new promise that resolves to an array of resolved values or reject when any one of them rejects. You may just want to get the results of all promises back and filter out the rejected ones:
const Fail = function(details){this.details=details;},
isFail = item => (item && item.constructor)===Fail;
Promise.all(
urls.map(//map array of urls to array of promises that don't reject
url =>
fetch(url)
.then(
undefined,//do not handle resolve yet
//when you handle the reject this ".then" will return
// a promise that RESOLVES to the value returned below (new Fail([url,err]))
err=>new Fail([url,err])
)
)
)
.then(
responses => {
console.log("failed requests:");
console.log(
responses.filter(//only Fail type
isFail
)
);
console.log("resolved requests:");
console.log(
responses.filter(//anything not Fail type
response=>!isFail(response)
)
);
}
);

promise me return a pending state when I use a callback

I test the fetch API with a callback, but my function returns "Promise State: Pending", and I don't understand why :
async function getJson(url, callback) {
await fetch(url)
.then(async function(response) {
return await response.json()
})
.then(function(data) {
console.log(data)
callback(data)
})
}
let getData = {
getAll: async function() {
await getJson('js/getJson/data.json', function(data) {
console.log(data.photographers) //OK
let test = data.photographers
return test
})
}
}
console.log(getData.getAll()); //return promise pending
Thanks
General advice for asynchronous and promise-based programming listed below...
Here's what getJson() should look like:
function getJson(url, callback) {
return fetch(url).then(function(response) {
return response.json();
});
}
Just return the promise that fetch() already returned - don't try to convert to a callback. Converting a promise-based interface to a callback interface is going backwards in terms of programming usability. Promises are much better to program with than plain callbacks for asynchronous programming.
Here's how getAll() can then use it:
const getData = {
getAll: async function() {
const data = await getJson('js/getJson/data.json');
return data.photographers;
}
}
Or, equally good (and equivalent to the above example) would be this:
const getData = {
getAll: function() {
return getJson('js/getJson/data.json').then(data => {
return data.photographers;
});
}
}
await has significant advantages when you are sequencing more than one asynchronous operation, but usually doesn't really help much when there's just one asynchronous operation. It's not wrong to use it then, it just doesn't really offer much help.
And, here's how one would call getAll():
getData.getAll().then(photographers => {
console.log(photographers);
}).catch(err => {
console.log(err);
});
General Advice and Explanation:
1. Read and study how async and await work with promises. Only use it once you understand that and then you will only be using await when you are awaiting a promise. await does nothing useful unless you are awaiting a promise so if you're awaiting a function call, then that function must be returning a promise and that promise must be connected to when the asynchronous operations in that function are completed.
2. Don't mix plain callbacks and promises. If you are programming with a promise interface, use that promise - never convert it to a plain callback. Return a promise from your function and let the caller use that. Among the many, many reasons that promises were invented is that control flow in non-simple asynchronous operations is massively simpler with promises (particularly asynchronous error handling and error propagation to higher levels).
3. Convert plain callbacks to promises. If you encounter an asynchronous operation that you want to use in a world where there are other promise-based asynchronous operations (such as fetch()), then wrap the plain callback into a promise interface so you are only mixing promise-based calls with other promise-based calls. Much, much simpler to code reliably.
4. async functions ALWAYS return a promise. That's how they are built internal to Javascript. So, a caller of an async function always gets back a promise as the return value. That promise will initially be in the pending state. It will be resolved sometime in the future and the eventual resolved value of the promise will be whatever value is returned from the outer scope of the async function. If there's no return statement in the outer scope of the async function, then the resolved value will be undefined as it is with both your `async functions.
5. A caller gets a resolved value from a promise with .then() or await. Those are the only two ways to get a resolved value out of a promise. So, any caller of an async function that wants some value back from it needs to use .then() or await to get that value.
6. If you have a promise-based operation inside a function and you wish to return it's resolved value from your function, then just return that promise from the function. That will allow the caller to use that promise to get the value. See my getJson() implementation above for how simple that can be.
7. Avoid return await fn() and use return fn() instead. If you're using return await fn(), then you're already in an async function and thus the function is already returning a promise. So, avoid the extra await as it doesn't do anything useful and just use return fn(). If fn() returns a value that value will become the resolved value of the promise that your async function returned. If fn() returns a promise, then the resolved value of that promise will become the resolved value of the promise that your async function returned.
8. Returning a value from a .then() handler becomes the resolved value of the parent promise. In the second getData() example above that uses .then() internally, the return data.photographers; statement sets the resolved value of the parent promise to data.photographers. So, any caller of getData() will find that data.photographers becomes the resolved value of the promise that getData() returns.
9. Returning a promise from a .then() handler chains the promises and the resolved value of the promise you return becomes the resolved value of the parent promise. Essentially, returning a promise from a .then() causes the parent promise to wait for the newly returned promise to resolve and it then gets its resolved value from that newly returned promise. You can see this in play in the getJson() function where response.json() returns a new promise that resolves to the json-parsed body of the http request. That resolved value will become the resolved value of the promise that the function returned.
10. Don't pass a callback when expecting a promise back. If you're passing a callback to some asynchronous function, then most of the time that function will not be returning a promise because many asynchronous APIs these days accept either a callback or return a promise, but don't do both at the same time. So, when looking to use await, make absolutely sure the function you're awaiting is returning a promise. When in doubt, look at the doc. If the doc is unclear look at the code for the function itself or run an experiment to see what the return value actually is. As an example, most of the mongodb asynchronous APIs will return a promise if you do NOT pass a callback to them, but will not return a promise if you do pass the callback. Use one or the other, not both.
It works, although it's not really what I wanted. Because I thought I could store my result outside of functions in a variable. But it seems that this is not possible in fact.
All this and due to the fact that I have to give an evaluation for my training. Fetch is not mandatory.
In the first version of my code, I didn't use it. I just did classes and function synchronously by loading my JSON into a variable and adding it to a script tag.
However, I wanted to do some tests because then I have to use the design pattern factory method. And I just tested and it works with your code. Thank you for taking the time to respond to me at such a late hour.
//we instantiate the factory
let factory = new identityFactory
getData.getAll().then(photographers => {
let identity = photographers
console.log(identity);
//we pass the identity in the factory
let newIdentity = factory.createIdentity(identity,'all')
console.log(newIdentity);
showIdentity(newIdentity)
}).catch(err => {
console.log(err);
});

How do I get data from pending resolved promise without async/await?

I have abstraction:
function fetchDataFromAPI() {
const url = `https://api...`
return fetch(url).then(response => response.json())
}
I want to use it in my other piece of code like:
if(something){
const data = fetchDataFromAPI()
return data
}
if I console.log data what I get is resolved pending promise
Promise {<pending>}
__proto__: Promise
[[PromiseStatus]]: "resolved"
[[PromiseValue]]: Object
How do I get that Object in data instead of Promise?
You can not. Here is why:
Promise is a language construct that makes JavaScript engine to continue to execute the code without waiting the return of inner function, also known as the executor function. A promise always run inside the event loop.
var p = new Promise(function(resolve, reject) {
setTimeout(function() {
resolve('foo');
}, 300);
});
console.log(p);
Basically a promise is a glorified syntactic sugar for a callback. We will see how but first lets have a more realistic code:
function someApiCall(){
return new Promise(function(resolve, reject){
setTimeout(()=>{
resolve('Hello');
})
})
}
let data = someApiCall();
console.log(data);
This is a so-called asynchronous code, when JavaScript engine executes it, someApiCall immediately returns a result, in this case pending promise:
> Promise {<pending>}
If you pay attention to the executor, you will see we needed to pass resolve and reject arguments aka callbacks. Yes, they are callbacks required by the language construct. When either of them called, promise will change its state and hence be settled. We don't call it resolved because resolving implies successful execution but a function also can error out.
How do we get the data? Well we need more callbacks, which will be called by the executor function once the promise is settled:
var p = new Promise(function(resolve, reject) {
setTimeout(function() {
resolve('foo');
}, 300);
});
p.then((result) => {
console.log(result); // foo
}).catch((err) => {
console.log(err);
});
Why we need to pass separate callbacks? Because one will be fed to the resolve, and the other to the reject. Then callback will be called by the resolve function, the catch callback by the reject function.
Javascript engine will execute these callbacks later on its leisure, for a regular function it means when the event loop is cleared, for timeout when the time is up.
Now to answer your question, how do we get data out from a promise. Well we can't.
If you look closely, you will see we don't really get the data out but keep feeding callbacks. There is no getting data out, but passing callbacks in.
p.then((result) => {
console.log(result);
}).catch((err) => {
console.log(err);
});
Some say use await:
async function() {
let result = await p;
}
But there is a catch. We have to or wrap it in async function. Always. Why? Because Async/await is another level of abstraction or syntactic sugar, whichever you prefer, on top of promise api. That is why we can not use await directly but always wrap it in async statement.
To sum up, when we use promise or async/await we need to follow certain convention and write terse code with closely knitted callbacks. Either javascript engine or transpilers like babeljs or typescript converts these code to regular javascript to be run.
I can understand your confusion because people keep saying getting data out when talking about promises, but we don't get any data out but pass callback to be executed when the data is ready.
Hope everything is clear now.
No, you cannot without using promises or async/await etc because calling a REST API is an asynchronous operation and is non blocking.
When you make a call to a REST API, the code shouldn't wait until the API returns a value because it may take a lot of time, making program non-responsive, thus by design making a network request is categorized as an asynchronous operation.
To avoid async/await, you'll need to use another .then:
if (something) {
return fetchDataFromAPI()
.then((data) => data /* you can console.log(data) here */)
}

Is there a way to end a thenable chain?

let textProcess = new Promise((resolve, reject) => {
let text
try {
// fetch text from the internet
text = "str"
resolve(text)
} catch (e) {
reject("failed to fetch!")
}
})
textProcess.then(data => {
let json
try {
json = JSON.parse(data)
} catch (e) {
console.error("failed to parse!")
// ..........i want to end the whole process here, never go ahead
}
}, e => {
// try to fetch text from local chache
}).then(json => {
// work on the json obj
}, e => {
// if failed to fetch from local too, just let it go
})
Is there a way to end a thenable chain?
Look at the example above, I want to end the whole process when parsing is failed(the line preceeded with ".........."). But actually the last then will still be invoked though.
What is the proper and elegant way to achieve my goal?
Your Promise usage involves quite a bit of sub-optimal patterns. Fixing them actually leads to what you're trying to achieve too.
textProcess.then(data => {
// 1st anti-pattern fix
// any error triggered here
// also gets caught at the end catch
return JSON.parse(data)
}).then(json => {
// work on json obj
}).catch(e => {
// 2nd anti-pattern fix
// one catch for the whole thenable chain
console.error("Failed to parse!", e)
})
This way, you properly leverage what Javascript Promise offers, and one simple .catch for what you need.
Edit - some explanations on involved Promise anti-patterns
The marked 1st anti-pattern is about unnecessary nested try..catch block within then. Within it, you can return synchronously (even undefined), another Promise (both of these are thenable), or throw an Error (which would get caught by catch). Basically you don't need to explicitly catch it but let it "flow" through.
The 2nd anti-pattern as mentioned is the fact that the second parameter -- reject handler of then is considered sub-optimal in most use cases. A Promise chain should be leveraging one catch to simplify the workflow.
However, in the rare event of the need to perform "early catch" and "resume", consider the following way, which is still a bit clearer than using two handlers for then:
textProcess.then(data => {
return parser1(data)
}).catch(e => {
console.log("First parser failed")
// for example first parser failed
return "fallback data"
}).then(data => {
// process received data as "fallback data"
}).catch(e => {
// always have a "last resort" catch at the end of the workflow
})
The short answer is no, there is no mechanism to terminate a promise chain in a then handler part-way down the chain. (A proposal for promise cancellation was made to the TC39 committee in 2016 but was subsequently withdrawn.)
Note that a "promise chain" often refers to the promise
returned by the last then, catch or finally call in a chain of single promise method calls concatenated together.
All calls to the promise methods listed above are made synchronously when code defining the chain is executed. After execution, all promises in the chain have been created and all method calls in the chain called.
Since promises only have three states (pending, fulfilled and rejected), the best you can do is to arrange for "cancellation" to be sent down the rejection channel of linked promises and ignore it as required. (There is no standard "cancelled" rejection value to use).

Return Promise result instead of Promise in Nodejs

Background
I am trying to learn promises, and I have a promise chain I want to improve on.
Problem
While learning how to chain promises, I fail to see why anyone would rather return a promise instead of returning it's value.
Take the following example, which uses promise chaining:
let myObj = new MyClass();
myObj.getInfo()
.then(result => writeOutput(FILE_NAME, result))
.then(console.log(FILE_NAME + " complete"))
.catch(error => console.error(error));
class MyClass{
getInfo() {
return new Promise(function(fulfil, reject) {
fulfill("I like bananas");
});
}
Here I have to chain 2 times. But if I were to directly return the result from the method getInfo() instead of returning a Promise I could potentially do something like the following:
let myObj = new MyClass();
let str = myObj.getInfo();
writeOutput(FILE_NAME, str)
.then(console.log(FILE_NAME + " complete"))
.catch(error => console.error(error));
Questions
So as you can see I am a little confused.
Given that getInfo() is in fact async, is it possible to achieve a similar code to the one in my second code sample?
If it were possible, would it be a good idea? How would you do it?
You can only return a value from some function if that value is immediately available when that function is called (on the same tick of the event loop). Remember that return is synchronous.
If it is not available right away then you can only return a promise (or you can use a callback but here you are specifically asking about promises).
For a more detailed explanation see this answer that I wrote some time ago to a question asking on how to return a result of an AJAX call from some function. I explained why you cannot return the value but you can return a promise:
jQuery: Return data after ajax call success
Here is another related answer - it got downvoted for some reason but I think it explains a similar issue that you're asking about her:
Return value in function from a promise block
I fail to see why anyone would rather return a promise instead of returning it's value.
Because you don't have the value yet.
Given that getInfo() is in fact async, is it possible to achieve a similar code to the one in my second code sample?
If it's asynchronous, it must return a promise.
A syntax that avoids then calls (but still uses and produces promises) is made possible by async/await:
async function writeInfo(FILE_NAME) {
const myObj = new MyClass();
try {
const str = await myObj.getInfo();
await writeOutput(FILE_NAME, str);
console.log(FILE_NAME + " complete");
} catch (error) {
console.error(error);
}
}
writeInfo("…");
The purpose of a promise is to say: "The data will be there sometime, just continue with your coding for now", so that your page won't get stuck for example. Basically you are using Promises for example to load something from the server.
Another way of looking at the first question ("why would anyone..."):
When you call .then or .catch on a promise, the pending promise returned from the call is appended to the end of an existing promise chain. (Allowing the "existing chain" might only contain one promise).
When you resolve (not "fulfill", you can't fulfill a promise with a promise) a promise with a promise, the promise used in resolution is inserted at the head of what remains of the promise chain.
The promise used in resolution must be fulfilled with a value before the next .then or .catch clause in the chain is called.

Categories