Imagine that we have an HTML page that fires AJAX requests. We want to make sure that AJAX requests are executed in order. The next AJAX request won't be fired until the previous one completes or errors.
I have tried to model this via a task queue using RxJS concatMap. Each AJAX request is modeled as an Observable. Everything is working great if AJAX request completes successfully, however if it errors, then the next task in the queue is not executed.
Here is an example, that uses setTimeout() to simulate long running async tasks:
function identity(observable) {
return observable;
}
function createTaskQueue() {
var subject= new Rx.Subject();
subject
.concatMap(identity)
.onErrorResumeNext(Rx.Observable.of('error'))
.subscribe(function(data) {
console.log('onNext', data);
},
function(error) {
console.log('onError', error);
});
return {
addTask: function(task) {
subject.next(task);
}
}
}
function createTask(data, delay) {
return Rx.Observable.create(function(obs) {
setTimeout(function() {
obs.next(data);
obs.complete();
}, delay);
});
}
function createErrorTask(data, delay) {
return Rx.Observable.create(function(obs) {
setTimeout(function() {
obs.error('Error: ' + data);
obs.complete();
}, delay);
});
}
var taskQueue = createTaskQueue();
taskQueue.addTask(createTask(11, 500))
taskQueue.addTask(createTask(22, 200));
taskQueue.addTask(createErrorTask(33, 1000));
taskQueue.addTask(createTask(44, 300));
taskQueue.addTask(createErrorTask(55, 300));
taskQueue.addTask(createTask(66, 300));
Here is an executable example: https://jsfiddle.net/artur_ciocanu/s6ftxwnf/.
When I run this code the following is printed to the console:
onNext 11
onNext 22
onNext error
Which is expected, but I wonder why the other tasks like 44, 55, etc are not executed.
I am pretty sure I am doing something stupid with onErrorResumeNext() or may be the whole approach is totally wrong.
Any help is very much appreciated.
If you read the documentation of onErrorResumeNext,
Continues an observable sequence that is terminated normally or by an
exception with the next observable sequence or Promise.
What that means is that when your source observable will encounter an error, it will switch to whatever you passed to onErrorResumeNext. What happens here is that Rx.of(...) terminates immediately after emitting its value. Hence the behavior you observe.
So in short, you don't want onErrorResumeNext here.
You could instead .catch(...) the stream which could emit an error. So, something like :
subject
.concatMap(obs => obs.catch(Rx.Observable.of('error')))
.subscribe(...)
the idea of an error in observables is the same as in regular function. Meaning if you throw an error in a regular function - function will not return anything. The same is with observables - if observable emits an error that means stream is completed and no more values is coming. So yes it is fundamentally wrong.
The better (right) approach would be to to have a stream of responses where next value can be either successful response or and error response. And if you need to separate them you can split responses stream into two successful/error responses later on.
Hope that helps.
Related
I would like to run this code with babel:
redisClientAsync.delAsync('key');
return await someOtherAsyncFunction();
inside an async function without await the first line. is this OK?
how else can I run something that I don't care?
Can I just fire the non-promisified function del('key',null) without a callback?
Yes, you can do that, and it will run the two asynchronous functions in parallel. You've just created a promise and thrown it away.
However, this means that when the promise is rejected you won't notice. You'll just get an unhandledRejection eventually which will crash your process if not handled.
Is this OK? How can I run something that I don't care?
Probably it's not OK. If you truly wouldn't care, you hadn't run it in the first place. So you should be clear and explicit what you care about (and what not):
do you want to wait? (for side effects)
do you need the result?
do you want to catch exceptions?
If you only want to wait and don't care for the result value, you can easily throw away the result:
void (await someAsyncFunction()); // or omit the void keyword,
// doesn't make a difference in an expression statement
If you don't care about exceptions, you can ignore them using
… someAsyncFunction().catch(function ignore() {}) …
You can throw that away, await it, do anything with it.
If you want the result, you have to await it. If you care about exceptions, but don't really want to wait, you may want to execute it in parallel with the following functions:
var [_, res] = await Promise.all([
someAsyncFunction(), // result is ignored, exceptions aren't
someOtherAsyncFunction()
]);
return res;
inside an async function without await the first line. is this OK?
Yes, there are cases where you'd want to do this which are perfectly reasonable. Especially where you don't care about the result - one example is an analytics tracking operation that should not interfere with business critical code.
how else can I run something that I don't care?
In many ways, however simply calling the promise function works. Your del without a callback would probably work in this case but some functions don't guard against not passing callbacks, so you can pass an empty function instead (.del('key', () => {})).
You do want to however make sure that you know about it failing, even if you don't want to disrupt the operation of code - so please consider adding a process.on("unhandledRejection', event handler to explicitly ignore these particular exceptions or suppress them via:
redisClient.delAsync('key').catch(()=>{});
Or preferably, something like:
redisClient.delAsync('key').catch(logErr);
From all the research I've made so far, I think it's fine to do it, as long as you guarantee that the function you are not awaiting for guarantees a way to handle its own errors in case that happens. For example, a try-catch wrapping the whole function body, like you see in the following snippet for the asyncFunction.
It doesn't matter if the function throws synchronously or asynchronously. It guarantees the your mainFunction will complete no matter what. That's the key point here.
If you don't guarantee that, you have to risks:
If it throws synchronously, your main function will not complete.
If it throws asynchronously, you'll get an unhandled excepction
// THIS IS SOME API CALL YOU DON'T WANT TO WAIT FOR
const mockAPI = () => {
console.log("From mockAPI");
return new Promise((resolve,reject) => {
setTimeout(() => reject("LATE THROW: API ERROR"), 500);
});
};
// THIS IS THE SOME ASYNC FUNCTION YOU CALL BUT NOT AWAIT FOR
const asyncFunction = async (syncThrow) => {
try {
console.log("Async function START");
if (syncThrow) throw new Error("EARLY THROW");
await mockAPI();
console.log("Async function DONE");
}
catch(err) {
console.log("From async function catch");
console.log(err.message || err);
return;
}
};
// THIS IS YOUR MAIN FUNCTION
const mainFunction = async (syncThrow) => {
try {
console.clear();
console.log("Main function START");
asyncFunction(syncThrow);
console.log("Main function DONE <<< THAT'S THE IMPORTANT PART");
}
catch(err) {
console.log("THIS WILL NEVER HAPPEN");
console.log(err);
}
};
<div>
<button onClick="mainFunction(true)">Sync throw</button>
<button onClick="mainFunction(false)">Async throw</button>
</div>
Not in Node.js.
Node does not wait for ever-pending Promises. If other tasks are already completed and there is nothing left in the event loop, the Node process will be terminated even though there exists pending promise.
For the following script, if someOtherAsyncFunction() get resolved in 5 seconds, but redisClientAsync.delAsync('key') takes 10 seconds to execute, the Node process will be terminated after 5 seconds in theory, before the first line is resolved.
async function doSomething() {
redisClientAsync.delAsync('key');
return await someOtherAsyncFunction();
}
await doSomething();
Setup: Because of the time limit on lambdas, I am using a lambda to process some work, then recursively call itself to continue the work, again and again until it is complete. It's just a chron job that I split up to make sure it won't timeout if the data is too large sometimes.
It looks roughly like this:
exports.handler = async (event, context, callback) => {
let response;
let err;
try {
const data = await getSomeAsyncDataToUpdate(); // this won't get previously updated data, so each recursion gets the first X data until there's none left
response = await updateSomeDataAsync(data);
if (!doneConditionIsMet(response)) {
callLambda(); // we do NOT await because the top level lambda would eventually timeout
}
} catch(e) {
err = e;
}
callback(err, response);
};
This is being called from a Cloudwatch Event (chron).
The Problem:
According to the docs, Cloudwatch Events are asynchronous calls and therefore, do not use the callback. https://docs.aws.amazon.com/lambda/latest/dg/nodejs-handler.html
For async handlers, you return a response, error, or promise to the runtime instead of using callback.
Here's the rub: if I omit the callback, the recursive lambda call doesn't happen. I can see that the function gets called, but it does not result in another lambda getting triggered and running.
Why?
I am out of ideas. The only thing I could come up with is that the lambda is exiting before the call actually executes, which would be wild because I assume the actual execution is synchronous. Also, if that were the case, I do not see how calling the callback would keep it alive long enough for that to finish.
If it is working by some fluke, is there a correct way to do this?
Notes on things I've already checked/tried:
the callback does not keep the lambda alive for a long time. It exits immediately as expected
I tried every version I could think of with removing the callback and returning something
As noted in the code snippet, I do NOT want to await the recursive call because that would mean the first lambda has to wait for them all to complete, which defeats the entire purpose of this
My requirement is like this
I want to run an axios call.
I don't want to block the code until it finished.
Also I don't want to know it's 200 or 500
This is my experiment code.
function axios() {
console.log("axios calling...");
return new Promise((resolve, reject) => {
setTimeout(function() {
resolve(console.log("DONE!"));
}, 10000);
});
}
function run() {
axios();
console.log("Before");
return;
console.log("This should never log");
}
run();
According to this experiment I think even though I return from the function still that promisified function is run. Which means it guaranteed call the axios.
My concern is,
if axios take 10 mins established connection with the API (NOT SEND THE POST REQUEST) if I return from the next line will axios wait that 10 mins and send the request or break the connection establishing when I return?
A promise will wait for data regardless of weather you call .then() on it or not (note that there are some exceptions[1] ). And the page will continue processing and wait for events as long as it's opened.
So if the request takes 10 minutes it will continue waiting (barring timeouts on either the server, gateway, router or browser) until you close the page/tab.
[1] Some libraries only triggers promise creation when you call .then() for example knex will keep returning a query object instead of a promise until you call .then() on the query object)
I am trying to poll a REST API to update a data table which is working fine with the following code:
pollData(url, interval) {
return Rx.Observable.interval(interval)
.mergeMap(() => this.http.get(url));
}
// get data
this.dataService.pollData(this.url, this.updateInterval)
.subscribe(
data => console.log(data),
err => console.log(err),
() => console.log('done'));
The problem is that error and complete never get called. Any suggestions to get this working with onError and onCompete would be greatly appreciated. Thanks!
About the onComplete call on the observer, it will be effected only when the source observable finishes. This means when the observable returned by pollData completes. As you are currently polling with no exit condition, then naturally your observable never completes.
To have this observable complete, you need to come up with an exit condition :
timeout (for instance, poll for X seconds, then stop polling)
number of polls
pollData-based condition (for instance, if no changes detected after X consecutive polling)
external completion signal
any other condition which makes sense to your use case
All these conditions are easy to implement with RxJS through they will require you to update the code of the pollData function.
For instance for the external completion signal, you could write :
// defining somewhere the subject for signalling end of polling
stopPollingS = new Rx.Subject();
// somehow pass this subject as a parameter of the polling function
pollData(url, interval, stopPollingS) {
return Rx.Observable
.interval(interval)
.mergeMap(() => this.http.get(url))
.takeUntil(stopPollingS);
}
// somewhere in your code when you want to stop polling
stopPollingS.onNext(true);
About the onError call on the observer, , I am not sure I get what is happening. Have you tried provoking an error and check the onError handler of your observer is indeed called? If there is no error, it is quite obvious that the onError will not be called.
Just in case anyone was wanting to know how I went about solving this problem and implemented the functionality that was required. Basically I just needed to wrap the observable in another and return the error as a data.
initiatePolling(url, interval) {
var http = this.http;
return Rx.Observable.create(function (observer) {
// initial request (no delay)
requestData();
var timerId = setInterval(requestData, interval);
function requestData() {
var subscription = http.get(url).timeout(20000)
.subscribe(
result => {
observer.next(result);
subscription.unsubscribe();
},
err => {
observer.next(err);
subscription.unsubscribe();
},
() => {
subscription.unsubscribe();
});
}
return function () {
observer.complete();
window.clearInterval(timerId);
}
});
}
When I create an observable from scratch, and have the observer error, then complete, the done part of the subscription never is invoked.
var observer = Rx.Observable.create(function(observer){
observer.onError(new Error('no!'));
observer.onCompleted();
})
observer.subscribe(
function(x) { console.log('succeeded with ' + x ) },
function(x) { console.log('errored with ' + x ) },
function() { console.log('completed') }
)
The output is:
errored with Error: no!
I'd expect it to be:
errored with Error: no!
completed
If I change the code to invoke onNext instead of onError, the observable properly completes:
var observer = Rx.Observable.create(function(observer){
observer.onNext('Hi!');
observer.onCompleted();
})
observer.subscribe(
function(x) { console.log('succeeded with ' + x ) },
function(x) { console.log('errored with ' + x ) },
function() { console.log('completed') }
)
I get the expected output:
succeeded with Hi!
completed
Why does it not complete when an error has occured?
That's because an error means completion, so the callback associated to onCompleted never gets called. You can review here Rxjs contract for observables (http://reactivex.io/documentation/contract.html) :
An Observable may make zero or more OnNext notifications, each representing a single emitted item, and it may then follow those emission notifications by either an OnCompleted or an OnError notification, but not both. Upon issuing an OnCompleted or OnError notification, it may not thereafter issue any further notifications.`
For error management, you can have a look at :
https://github.com/Reactive-Extensions/RxJS/blob/master/doc/gettingstarted/errors.md
Another and probably the simplest solution might be using the add() function.
The statement will be always executed regardless an error
occured or not (like the finally statement in most programming languages).
observer.subscribe(
function(x) { console.log('succeeded with ' + x ) },
function(x) { console.log('errored with ' + x ) },
function() { console.log('completed') }
)
.add(() => {
console.log("Will be executed on both success or error of the previous subscription")
);
While I was having the same question, I bumped into this github issue.
Apparently finally method of Observable object needs to be used in this case.
Quoting from Aleksandr-Leotech from that thread:
Complete and finally are totally different things. Complete means that
the observable steam was finished successfully. Because you can have
many success calls. Finally means that steam has ended, either
successfully or not.
It is not obvious with HTTP requests, but imagine two additional
scenarios.
Mouse events. You will be receiving a never-ending steam of success
callbacks, but you will never receive finally or complete, because
user events will never stop (unless you trigger an exception with
buggy code, then you will get error and finally).
Working with web sockets. You will get multiple success callbacks, but at some point in time your communication with back end will stop and you will get both complete and finally unless you have some errors, which will call error and finally.
So, you might be getting multiple or no success calls, zero or one error call, zero or one complete and zero or one finally.
To run a callback when observable completes or errors, you should use finalize.
Ex:
this.service.yourObservable
.pipe(
finalize(() => {
// * This will always run when observable finishes the stream
console.log("Finally!");
// * callback for finally
})
).subscribe(
{
next: () => { // * Callback for success },
error: () => { // * Callback for error },
complete: () => {// * This gets called only on success }
})