Just like How to avoid "if" chains?, this question is about what should be done to keep asynchronous programs clean.
Asynchronous programming is a totally different paradigm from procedural programming, and its really fun when you get your head around it. However, it can get messy when you have several asynchronous calls.
Consider this example (This is an abstraction of a real world scenario I encountered):
Read file a.txt
Submit contents of a.txt to server X for processing and retrieve the output.
Read file b.txt.
Send contents of b.txt, and the response from server X (step 2) to server Y, and get the output.
Save the response from server Y (step 4) to c.txt.
With asynchronous calls, my JavaScript looks similar to this (function names are made up. All functions are asynchronous calls with callbacks. Error parameters are omitted to improve clarity. Real code with error parameters and error handling is a lot messier than this):
readFile('a.txt', function (file_a_data) {
requestToServer('http://xserver.com/', file_a_data, function (server_x_response) {
readFile('b.txt', function (file_b_data) {
var request_params = server_x_response + file_b_data;
requestToServer('http://yserver.com/', request_params, function (server_y_reponse) {
writeFile('c.txt', server_y_response);
});
});
});
});
As you can see, there is already four levels of indentation, and an arrow of code is building up. How can we avoid this nesting of callbacks, and write some clean code?
What I have tried so far:
One way of doing this would be to write synchronous, no-callback versions of the functions, whenever possible. A value is returned and on error exception is thrown.
try {
var file_content = readFile('a.txt');
// do stuff with file_content
} catch (ex) {
// an error occured
}
But there are few problems with this:
For I/O heavy programs like this, the performance hit is very high.
Elementary API functions (such as XmlHttpReqeust, or Node.js File System API already have callbacks. They are the asynchronous parts, and there is little we can do other than writing thin, synchronous wrappers.
Making all callbacks named functions and just specifying the function name for the callback:
function processFileA(data) {
requestToServer('http://xserver.com/', file_a_data, processRequestX);
}
function processRequestX(response) {
readFile('b.txt', function (file_b_data) {
var request_params = server_x_response + file_b_data;
requestToServer('http://yserver.com/', request_params, processRequestY);
});
}
function processRequestY(response) {
writeFile('c.txt', server_y_response);
}
readFile('a.txt', processFileA);
This is similar to How to avoid hard-coded, chained asynchronous functions in Javascript/jQuery? This question looks like the inversion of my problem. The coding style of this question was one remedy I used to stop chains, but it does not look very nice. I think it looks like thin wrappers and spaghetti:
how to avoid callback chains?: A similar question (in title), but this is not really about asynchronous callbacks. This is about passing functions as parameters.
Use Promises! That's exactly what they were made for:
// assume 'readFile' and 'requestToServer' return a promise for their result
readFile('a.txt').then(function(file_a_data) {
return requestToServer('http://x.example.com/', file_a_data);
}).then(function(server_x_response) {
return readFile('b.txt').then(function(file_b_data) {
return server_x_response + file_b_data;
});
}).then(function(request_params) {
return requestToServer('http://y.example.com/', request_params);
}).then(function(server_y_reponse) {
writeFile('c.txt', server_y_response);
});
(Yes, it stays as clean as this, there is no additional, messy error handling code)
It might get even more clean if you leverage a bit of functional programming and use curried functions, or at least partially apply them. Even reading a and b in parallel would be trivially achievable:
Promise.all([
readFile('a.txt').then(requestToServer.partial('http://x.example.com/')),
readFile('b.txt')
])
.spread(function(server_x_response, file_b_data) {
return server_x_response + file_b_data;
})
.then(requestToServer.partial('http://y.example.com/'))
.then(writeFile.partial('c.txt'));
Related
Express documentation Production best practices: performance and reliability says:
Don’t use synchronous functions
Synchronous functions and methods tie up the executing process until
they return. A single call to a synchronous function might return in a
few microseconds or milliseconds, however in high-traffic websites,
these calls add up and reduce the performance of the app. Avoid their
use in production.
So my question is, in the context of node/express, if I have a function that accepts some static value and returns a calculated result (what I would normally consider a "synchronous function"), is it best practice to wrap that function inside a new Promise and resolve the result or does this create any significant unnecessary overhead? For example:
Current:
//inside my index.js
var myArgument = 'some long string';
var myResult = myFunction(myArgument);
function myFunction(myArgument){
var thisResult;
//some calculations
return thisResult;
}
New (and Improved?)
//inside my index.js
(async function() {
var myArgument = 'some long string';
var myResult = await myFunction(myArgument);
});
function myFunction(url) {
return new Promise((resolve, reject) => {
var thisResult;
//some calculations
if(thisResult){
resolve (thisResult);
} else {
reject (null)
}
});
}
Short answer: Nope.
The documentation is talking about not using synchronous versions of functions like readfileSync from nodeJS filesystem or bcrypt.compareSync for example. Synchronous calls block the event loop in nodeJS. So nothing happens while you are waiting for the synchronous call to finish. The whole program is on halt while this one method finishes. This is bad in a single threaded system like nodeJS.
There no reason to wrap functions that are just simple calculations or array manipulations with callbacks or promises.
Its just saying that if there's a library/method that offers synchronous version of the method, try to avoid that method.
Check out: https://nodejs.org/en/docs/guides/blocking-vs-non-blocking/
JavaScript execution in Node.js is single threaded, so concurrency
refers to the event loop's capacity to execute JavaScript callback
functions after completing other work. Any code that is expected to
run in a concurrent manner must allow the event loop to continue
running as non-JavaScript operations, like I/O, are occurring.
As an example, let's consider a case where each request to a web
server takes 50ms to complete and 45ms of that 50ms is database I/O
that can be done asynchronously. Choosing non-blocking asynchronous
operations frees up that 45ms per request to handle other requests.
This is a significant difference in capacity just by choosing to use
non-blocking methods instead of blocking methods.
The event loop is different than models in many other languages where
additional threads may be created to handle concurrent work.
Regarding additional overhead with wrapping everything in promises. The answer is still no.
You will experience no difference in
function sum(x,y) {
return x+y
}
const ans = sum(1,2)
console.log(ans) // 3
and
function sum(x,y) {
return Promise.resolve(x+y) // Shorthand for your new Promise
}
sum(1,2).then(ans => {
console.log(ans) //3
})
I decided to have a fallback call to the server when creating a model in case we can't find a value from the information available locally. I want to have it async if possible, does it mean I have to make everything in this chain of code consume and return promises?
I'm going to use pseudo code to illustrate this point initially, as the call stack can be a little long and so we don't include things that aren't central to this question. Let me know if you need more to go on though.
function router() {
if (condition) {
controller.renderDocument(options);
}
}
controller.renderDocument(options) {
documentCollection.findOrCreateDocumentHolder(options);
}
documentCollection.findOrCreateDocumentHolder(options) {
var model = this.findOrCreateDocument(options);
}
documentCollection.otherFunction(options) {
// just to illustrate that there's several entry points to this kind of functionality
var model = this.findOrCreateDocument(options);
}
documentCollection.findOrCreateDocument(options) {
if (!options.type)
options.type = resolveType(options);
// other things that use the found options.type
return model;
}
resolveType(options) {
var locallyResolved = resolveLocally(options);
if (locallyResolved)
return locallyResolved;
// try the server as a fallback
var serverResolved = resolveFromServer(options.id);
return serverResolved;
}
resolveLocally(options) {
return stuff;
}
resolveFromServer(id) {
return ajaxRequestResult();
}
Obviously the actual code is a bit longer and actually does stuff.
It feels excessive to make all of this use chaining promises for the sake of 1 possible ajax request down the pipe, especially as this will be a rare occurrence. The other alternative is to just make a synchronous AJAX request.
Are my only options to make everything use promises and whens, or to make it a synchronous AJAX request? Is there a nicer alternative? I'm thinking of C#'s await keyword, but I'm sure other languages have similar functionality.
I want to have it async if possible, does it mean I have to make everything in this chain of code consume and return promises?
Yes. Something that is potentially asynchronous must always return a promise, and must always be consumed as if it was asynchronous (promises enforce this anyway).
Is the only other alternative to just make a synchronous AJAX request?
Yes, and you really don't want that.
Is there a nicer alternative? I'm thinking of C#'s await keyword, but I'm sure other languages have similar functionality.
In fact, async-await is coming to ECMAScript as well. But it doesn't make anything synchronous, it rather is just syntactic sugar for promises-everywhere.
Once you start with promises they do spread around, no way around it really since once you go async you must either wait (not feasible) or pass callbacks (which promises solve).
My approach is to embrace them, you'll be more often returning promises then immediate values, but you'll always take care of handling success/error and have a consistent way of dealing with data flow.
There are many more sources of asynchronicity than AJAX requests and I really can't think of a case where I would require a synchronous return value in Javascript.
I have a function,
Edit1 - Updated function with real one because the previous one was simplified synchronous function and the code would have worked as correctly pointed by #AlexMA in the comments
'returnSuccessOrFailure': function () {
return driver.findElement(wd.By.css('div#button')).then(function (button) {
return button.getAttribute('class').then(function (status) {
return status;
});
});
}
In my node.js test, my assertion is failing because the assert is called before returnSuccessOrFailure finishes execution.
var value = returnSuccessOrFailure();
assert.equal(value,'success', 'Looks like something failed');
If I implement a promise in returnSuccessOrFailure and chain my assert then that works. My question is do I have to implement promises all the time for such situations to block the execution? I am new to Javascript and the async nature of it and any insight when to use promises and when not to would be useful.
you don't have to "implement a promise" in, just return the one you already have:
returnSuccessOrFailure': function () {
return driver.findElement(wd.By.css('div#button')).then(function (button) {
...
but then, yes, you do still need to put your assert in a done handler
returnSuccessOrFailure().done(function(value) {
assert.equal(value,'success', 'Looks like something failed');
}
Chaining you asserts will not only make it work but will also make for more readable code. Knowing what happens in what order can be useful when going back to refactor. Not only that but the structure of callbacks/promises also allow for easily written timer tests.
Also, since your test needs to have the current state of execution, it is more than likely that writing tests with asserts in callbacks is what you will need anyway.
My question is do I have to implement promises all the time for such situations to block the execution?
Notice that promises don't block the execution. They defer the execution of code that depends on the result, notice that you're still chaining callbacks on them.
I am new to Javascript and the async nature of it and any insight when to use promises and when not to would be useful.
Promises are useful wherever you have some code that might run asynchronously and needs to pass back an asynchronous result. Otherwise you would need to use callbacks, which are way more ugly than promises.
This is part of code contracts and representing preconditions (what holds before you execute), postconditions (what holds after you execute), and object invariants (what can not change). JavaScript does not have native support for this, but you can use third party libraries (Cerny.js, ecmaDebug, jsContract, or jscategory)
I think it depends on your coding style, is it EAFP(Easier to ask for forgiveness than permission) or LBYL(Look before you leap). Both are viable! In most compiled languages you would use LBYL. However in Python for example you would use EAFP.
Generally if you know you will fail you want to fail fast. If you like to use assertions to ensure code fails fast it is up to you.
I'm not so experienced/elegant programmer. I hope my question is understandable.
I have used java/c++ for most part of my life so my mind is object oriented. Then I learned python and I enjoyed very much functional mind. Now I'm approaching js. It is challenging because it is function based (and I liked it a lot), functional (with underscore I have a lot of pythonic iterutils methods) and asynchronous (it hurts).
A lot of time I have some back-end sync procedural flow like:
if exist(user):
foo(user)
# ok
else:
add(user)
# user added
Now I have to handle this with the so called callback hell:
exists(user, function(userExist) {
if( userExist ) {
foo( user, function(fooResult) {
/* ok */
});
} else {
add( user, function(addResult) {
/* user added */
});
}
});
Sometimes the controls are more nested: check for token, if token is valid check for user exists, if user exists check for valid parameters, then check for no error in foo on user, etc…
Those controls are simply synchronous and imperative-like. Nothing else to say. While with sync language like python I can handle this is (not elegant but at least) readable code with the use of small function that returns values , with javascript I don't know how to refactor things in readable small function. All functions I wrote does not have any return statements but just callback(something_weird_defined_in_caller_function) and I lost myself.
I don't think that promises are good on my case because are more for piping events, IIUC. So I'm looking for some pattern using async library to handle this case:
- continue to execute functions in series only if the previous one succeeded.
Please any help will be appreciated.
I don't think that promises are good on my case because are more for piping events, IIUC
No. Promises don't represent a stream of events to pipe, but a single result that will arrive later (asynchronously). This is exactly what they were made for, and they will give you imperative-like looking code (instead of a nesting callback hell).
Admittedly, you still need nesting for control structures. The code that you presented won't look much different except for the callbacks going into .then() calls, and promises being returned.
However, you can also use exceptions for control flow. Instead of using exists that returns a Promise<boolean>, you could use a find function that returns a Promise<User> which would get rejected when the user doesn't exist. It could be used like
find(user).then(function(/*user*/) {
return foo(user) /* .then(function(fooResult) {
…
}); */
}).catch(UserNotFoundError, function(e) {
return add(user) /* .then(function(addResult) {
…
}); */
});
Which of those to choose is debatable, rejections should be exceptional.
I have developed a small lib for the Dynamics CRM REST/ODATA webservice (CrmRestKit). The lib dependes on jQuery and utilizes the promise-pattern, repectivly the promise-like-pattern of jQuery.
Now I like to port this lib to bluebird and remove the jQuery dependency. But I am facing a problem because bluebird does not support the synchronous resolution of promise-objects.
Some context information:
The API of the CrmRestKit excepts an optional parameter that defines if the web-service call should be performed in sync or async mode:
CrmRestKit.Create( 'Account', { Name: "foobar" }, false ).then( function ( data ) {
....
} );
When you pass "true" or omit the last parameter, will the method created the record in sync. mode.
Sometimes it is necessary to perform a operation in sync-mode, for instance you can write JavaScript code for Dynamics CRM that is involed for the save-event of an form and in this event-handler you need to perform sync-operation for validation (e.g. validate that a certain number of child-records exist, in case the right number of records exist, cancel the save-operation and show an error message).
My problem now is the following: bluebird does not support the resolution in sync-mode. For instance when I do the following, the "then" handler is invoked in async fashion:
function print( text ){
console.log( 'print -> %s', text );
return text;
}
///
/// 'Promise.cast' cast the given value to a trusted promise.
///
function getSomeTextSimpleCast( opt_text ){
var text = opt_text || 'Some fancy text-value';
return Promise.cast( text );
}
getSomeTextSimpleCast('first').then(print);
print('second');
The output is the following:
print -> second
print -> first
I would expect that the "second" appears after the "first" because the promise is already resolved with an value. So I would assume that an then-event-handler is immediately invoked when applied on an already resolved promise-object.
When I do the same (use then on an already resolved promise) with jQuery I will have my expected result:
function jQueryResolved( opt_text ){
var text = opt_text || 'jQuery-Test Value',
dfd = new $.Deferred();
dfd.resolve(text);
// return an already resolved promise
return dfd.promise();
}
jQueryResolved('third').then(print);
print('fourth');
This will generate the following output:
print -> third
print -> fourth
Is there a way to make bluebird work in the same fashion?
Update:
The provided code was just to illustrate the problem. The idea of the lib is: Regardless of the execution-mode (sync, async) the caller will always deal with an promise-object.
Regarding "... asking the user... doesn't seems to make any sense": When you provide two methods "CreateAsync" and "CreateSync" it is also up to the user to decide how the operation is executed.
Anyway with the current implementation the default behavior (last parameter is optional) is a async execution. So 99% of the code requires a promise-object, the optional parameter is only use for the 1% cases where you simply need a sync execution. Furthermore I developed to lib for myself and I use in 99,9999% of the case the async mode but I thought it is nice to have the option to go the sync-road as you like.
But I thinks I got the point an sync method should simply return the value. For the next release (3.0) I will implement "CreateSync" and "CreateAsync".
Thanks for your input.
Update-2
My intension for the optional parameter was to ensure a consistend behavior AND prevent logic error. Assume your as a consumer of my methode "GetCurrentUserRoles" that uses lib. So the method will alway return an promise, that means you have to use the "then" method to execute code that depends on the result. So when some writes code like this, I agree it is totally wrong:
var currentUserRoels = null;
GetCurrentUserRoles().then(function(roles){
currentUserRoels = roles;
});
if( currentUserRoels.indexOf('foobar') === -1 ){
// ...
}
I agree that this code will break when the method "GetCurrentUserRoles" changes from sync to async.
But I understand that this I not a good design, because the consumer should now that he deals with an async method.
Short version: I get why you want to do that, but the answer is no.
I think the underlying question being asked is whether a completed promise should immediately run a callback, if the promise has already completed. I can think of a lot of reasons that this might happen - for example, an asynchronous save procedure that only saves data if changes were made. It may be able to detect changes from the client side in a synchronous fashion without having to go through an external resource, but if changes are detected then and only then would an asynchronous operation be required.
In other environments that have asynchronous calls, the pattern seems to be that the developer is responsible for understanding that their work might complete immediately (for example, .NET framework's implementation of the async pattern accomodates this). This is not a design problem of the framework, it's the way it's implemented.
JavaScript's developers (and many of the commenters above) seem to have a different point of view on this, insisting that if something might be asynchronous, it must always be asynchronous. Whether this is "right" or not is immaterial - according to the specification I found at https://promisesaplus.com/, item 2.2.4 states that basically no callbacks can be called until you are out of what I'll refer to as "script code" or "user code"; that is, the specification says clearly that even if the promise is completed you can't invoke the callback immediately. I've checked a few other places and they either say nothing on the topic or agree with the original source. I don't know if https://promisesaplus.com/ could be considered a definitive source of information in this regard, but no other sources that I saw disagreed with it and it seems to be the most complete.
This limitation is somewhat arbitrary and I frankly prefer the .NET perspective on this one. I'll leave it up to others to decide if they consider it "bad code" to do something that might or might not be synchronous in a way that looks asynchronous.
Your actual question is whether or not Bluebird can be configured to do the non-JavaScript behavior. Performance-wise there may be a minor benefit to doing so, and in JavaScript anything's possible if you try hard enough, but as the Promise object becomes more ubiquitous across platforms you will see a shift to using it as a native component instead of custom written polyfills or libraries. As such, whatever the answer is today, reworking a promise in Bluebird is likely to cause you problems in the future, and your code should probably not be written to depend on or provide immediate resolution of a promise.
You might think this is a problem, because there's no way to have
getSomeText('first').then(print);
print('second');
and to have getSomeText "first" printed before "second" when the resolution is synchronous.
But I think you have a logic problem.
If your getSomeText function may be synchronous or asynchronous, depending on the context, then it shouldn't impact the order of execution. You use promises to ensure it's always the same. Having a variable order of execution would likely become a bug in your application.
Use
getSomeText('first') // may be synchronous using cast or asynchronous with ajax
.then(print)
.then(function(){ print('second') });
In both cases (synchronous with cast or asynchronous resolution), you'll have the correct execution order.
Note that having a function being sometimes synchronous and sometimes not isn't a weird or unlikely case (think about cache handling, or pooling). You just have to suppose it's asynchronous, and all will be always fine.
But asking the user of the API to precise with a boolean argument if he wants the operation to be asynchronous doesn't seem to make any sense if you don't leave the realm of JavaScript (i.e. if you don't use some native code).
The point of promises is to make asynchronous code easier, i.e. closer to what you feel when using synchronous code.
You're using synchronous code. Don't make it more complicated.
function print( text ){
console.log( 'print -> %s', text );
return text;
}
function getSomeTextSimpleCast( opt_text ){
var text = opt_text || 'Some fancy text-value';
return text;
}
print(getSomeTextSimpleCast('first'));
print('second');
And that should be the end of it.
If you want to keep the same asynchronous interface even though your code is synchronous, then you have to do it all the way.
getSomeTextSimpleCast('first')
.then(print)
.then(function() { print('second'); });
then gets your code out of the normal execution flow, because it's supposed to be asynchronous. Bluebird does it the right way there. A simple explanation of what it does:
function then(fn) {
setTimeout(fn, 0);
}
Note that bluebird doesn't really do that, it's just to give you a simple example.
Try it!
then(function() {
console.log('first');
});
console.log('second');
This will output the following:
second
first
There are some good answers here already, but to sum up the crux of the matter very succinctly:
Having a promise (or other async API) that is sometimes asynchronous and sometimes synchronous is a bad thing.
You may think it's fine because the initial call to your API takes a boolean to switch off between sync/async. But what if that's buried in some wrapper code and the person using that code doesn't know about these shenanigans? They've just wound up with some unpreditable behavior through no fault of their own.
The bottom line: Don't try to do this. If you want synchronous behavior, don't return a promise.
With that, I'll leave you with this quotation from You Don't Know JS:
Another trust issue is being called "too early." In application-specific terms, this may actually involve being called before some critical task is complete. But more generally, the problem is evident in utilities that can either invoke the callback you provide now (synchronously), or later (asynchronously).
This nondeterminism around the sync-or-async behavior is almost always going to lead to very difficult to track down bugs. In some circles, the fictional insanity-inducing monster named Zalgo is used to describe the sync/async nightmares. "Don't release Zalgo!" is a common cry, and it leads to very sound advice: always invoke callbacks asynchronously, even if that's "right away" on the next turn of the event loop, so that all callbacks are predictably async.
Note: For more information on Zalgo, see Oren Golan's "Don't Release Zalgo!" (https://github.com/oren/oren.github.io/blob/master/posts/zalgo.md) and Isaac Z. Schlueter's "Designing APIs for Asynchrony" (http://blog.izs.me/post/59142742143/designing-apis-for-asynchrony).
Consider:
function result(data) {
console.log( a );
}
var a = 0;
ajax( "..pre-cached-url..", result );
a++;`
Will this code print 0 (sync callback invocation) or 1 (async callback invocation)? Depends... on the conditions.
You can see just how quickly the unpredictability of Zalgo can threaten any JS program. So the silly-sounding "never release Zalgo" is actually incredibly common and solid advice. Always be asyncing.
What about this case, also CrmFetchKit related which in latest version uses Bluebird. I have upgraded from version 1.9 that was based on jQuery. Still the old app code that uses CrmFetchKit has methods the prototypes of which I can't or won't change.
Existing App Code
CrmFetchKit.FetchWithPaginationSortingFiltering(query.join('')).then(
function (results, totalRecordCount) {
queryResult = results;
opportunities.TotalRecords = totalRecordCount;
done();
},
function err(e) {
done.fail(e);
}
);
Old CrmFetchKit implementation (a custom version of fetch())
function fetchWithPaginationSortingFiltering(fetchxml) {
var performanceIndicator_StartTime = new Date();
var dfd = $.Deferred();
fetchMore(fetchxml, true)
.then(function (result) {
LogTimeIfNeeded(performanceIndicator_StartTime, fetchxml);
dfd.resolve(result.entities, result.totalRecordCount);
})
.fail(dfd.reject);
return dfd.promise();
}
New CrmFetchKit implementation
function fetch(fetchxml) {
return fetchMore(fetchxml).then(function (result) {
return result.entities;
});
}
My problem is that the old version had the dfd.resolve(...) where I was able to pass any number of params that I need.
The new implementation just returns, the parent seems to call the callback, I can't call it directly.
I went and made a custom version of the fetch() in the new implementation
function fetchWithPaginationSortingFiltering(fetchxml) {
var thePromise = fetchMore(fetchxml).then(function (result) {
thePromise._fulfillmentHandler0(result.entities, result.totalRecordCount);
return thePromise.cancel();
//thePromise.throw();
});
return thePromise;
}
But the problem is that the callback gets called two times, once when I do it explicitly and second by the framework but it passes it one parameter only. To trick it and "tell" not to call anything because I do it explicitly I try to call .cancel() but it is ignored. I understood why but still how do you do the "dfd.resolve(result.entities, result.totalRecordCount);" in the new version with out having to changes prototypes in the app that uses this library ?
You can in fact do this, yes.
Modify the bluebird.js file (for npm: node_modules/bluebird/js/release/bluebird.js), with the following change:
[...]
target._attachExtraTrace(value);
handler = didReject;
}
- async.invoke(settler, target, {
+ settler.call(target, {
handler: domain === null ? handler
: (typeof handler === "function" &&
[...]
For more info, see here: https://github.com/stacktracejs/stacktrace.js/issues/188