Node.JS - Q Module - Promises - javascript

I'm hoping someone who uses the Q module in Node.js can advise me.
So, my question is:
Is it typically necessary to create more than one promise if you are trying to perform multiple functions in sequence?
For example, I'm trying to create an application that:
1) Read data from a file (then)
2) Opens a database connection (then)
3) Executes a query via the database connection (then)
4) Store JSON dataset to a variable (then)
5) Close Database connection. (then)
6) Perform other code base on the JSON dataset I've stored to a variable.
In raw Node.js, each method of an object expects a callback, and in order to do these tasks in the proper order (and not simultaneously -which wouldn't work), I have to chain these callbacks together using an ugly amount of code nesting.
I discovered the Q module, which prevents nesting via the Promise concept.
However, in my first attempt at using Q, I'm trying to make a promise out of everything, and I think I might be over-complicating my code.
I'm think that maybe you only really have to create one promise object to perform the steps mentioned above, and that it may be unnecessary for me to convert every method to a promise via the Q.denodeify method.
For example, in my code I will be connecting to a db2 database using the ibm_db module. Probably due to misunderstanding, I've converted all the ibm_db methods to promises like this:
var ibmdb = require('ibm_db');
var q = require('q');
var ibmdbOpen = q.denodeify(ibmdb.open);
var ibmdbConn = q.denodeify(ibmdb.conn);
var ibmdbClose = q.denodeify(ibmdb.close);
var ibmdbQuery = q.denodeify(ibmdb.query);
Is this really necessary?
In order to do one thing after another in Q, is it necessary for me to denodeify each method I'll be using in my script?
Or, can I just create one promise at the beginning of the script, and use the q.then method to perform all the asynchronous functions in sequence (without blocking).

Is it typically necessary to create more than one promise if you are trying to perform multiple functions in sequence?
Yes, definitively. If you didn't have promises for all the intermediate steps, you'd have to use callbacks for them - which is just what you were trying to avoid.
I'm trying to make a promise out of everything
That should work fine. Indeed, you should try to promisify on the lowest possible level - the rule is to make a promise for everything that is asynchronous. However, there is no reason to make promises for synchronous functions.
Especially your steps 4 and 5 trouble me. Storing something in a variable is hardly needed when you have a promise for it - one might even consider this an antipattern. And the close action of a database should not go in a then handler - it should go in a finally handler instead.
I'd recommend not to use a linear chain but rather:
readFile(…).then(function(fileContents) {
return ibmdbOpen(…).then(function(conn) {
return imbmdbQuery(conn, …).finally(function() {
return imbdbClose(conn);
});
}).then(function(queriedDataset) {
…
});
});

Related

How to control C# Task (async/await in same way as javascript Promise)?

I'm js developer and jumped to C#. I'm learning async/await/Task in C# and can't understand how to control C# Task.
In javascript I can save Promise resolve handler to call it later (and resolve promise), for example, to organize communication between some "badly connected" app parts (request doesn't returns result, but result is sent separately and fires some "replyEvent"), and do something like this:
// javascript
const handlers = {}
// event firing on reply
const replyEventListener = (requestId, result) => {
if (handlers[requestId]) {
handlers[requestId](result)
delete handlers[requestId]
}
}
const requestFunction = (id, params) => new Promise(resolve => {
// handler is saved to use later from "outside"
handlers[id] = resolve;
// do required request
makeRequest(params)
})
// .. somewhere in code request becomes simple awaitable
const resultOfRequest = await requestFunction(someUniqueId)
How can I do same with C# Task?
I think the philosophy behind Promise (in JS) and Task (in C#) is different.
We don't implement them exactly the same way.
Read the documentation on the MSDN :
Task Class - Remarks
Task<TResult> Class
You must empty your cup to be able to fill it again.
A good article in MSDN is the following which shows examples of how Tasks can be used like Promises in the sense that you can start them and then await them later.
Asynchronous programming with async and await
If you want to await a Task in C# like you would a Promise in JS, you would do the following:
Create a method or function that returns a Task. Usually this will be a Task of some class object. In the article above, it references creating breakfast as an analogy, so you can define FryEggsAsync as a method that takes in an int number of how many eggs to fry and returns a Task<Egg>. The C# convention is to end any function or method name with Async to indicate that a Task is being returned.
Create your Task, which is similar to a Promise in JS, so you could do var eggsTask = FryEggsAsync(2), which would store a Task<Egg> in eggsTask, that you can then await or pass to other functions as needed.
To await your Task to resolve it and get the result, you would simply do await eggsTask.
You can also use Task.WaitAll to await multiple Tasks at once, similar to Promise.all in JS, by passing in an array of Tasks as an argument to that method call. See the Task.WaitAll documentation for some examples of this. The difference here though is that the results are stored separately in each task passed to Task.WaitAll, instead of aggregated together into a results array like in JS with Promise.all, but if you need to await multiple Tasks then this will be an easier way to do so.

JavaScript initializing callback parameters down the callback chain

Note: I'm bootstrapping a reactjs app but this is a general JavaScript question.
I have a special module "locationUtils" that I am trying to keep in it's own package, but keeping that code separate is causing an eyesore with callbacks.
When I access one of it's methods I have to send a callback with it that only has one of its parameters initially defined, and in that method I get the other data parameter to initalize the other parameter.
Can I add in undefined parameters later like that in JavaScript, and is it good practice to initial parameters for a callback method as you go down the callback chain in general, or am I making a convoluted newbie mistake?
/********************Module 1******************************/
var bootStrapUI = function(callback) {
locationUtils.findData(findOtherData(callback));
}
//This gets called last to finalize bootstraping
var findOtherData = function(callback,originalFetchedData){
//use originalFetchedData to get more data
//bootStraping program with all rendering data
callback() //sends back a boolean confirming all fetched
}
/**********************Module2**********************************/
var findData = function(findOtherData){
var data = magicGetData();
findOtherData(findOtherData,data);//I initialized a param late here!
}
It's a good Javascript question, callbacks can become a serious hell for the uninitiated, particularly when they are nested and / or the order in which they return is important.
This is where promises come in: they are an essential tool for Javascript development and about to become part of the standard (in EcmaScript 6).
In essence: a promise is an object that is returned from a function with a method (callback) that is called when the asynchronous action (e.g. API call) has been completed. The difference between a promise and a callback is that promises allow you to structure how you handle the callbacks and, importantly, in what order.
I recently wrote a method that had to make 30 api calls with each call dependent on the results of the previous one (this was not a well designed api). Can you imagine trying to do that with callbacks? As it was, I created an array of promises and used jQuery.when() to handle things when all the api calls had completed.
For the moment we need to use a library for promises. jQuery: https://api.jquery.com/jquery.deferred/ is the obvious one but there are various other implementations that do much the same thing.
Update:
The question relates more specifically to the passing of arguments between callbacks and modifying the arguments as execution moves between them. This can be done easily by passing whatever info you need as an argument to your resolve method.
Typically it looks something like this (using jQuery):
var myAsyncMethod = function(info){
var deferred = $.Deferred();
$.getJSON(myUrl,
function(dataFromServer) {
// Do stuff with data
var newData = doSomething(dataFromServer);
deferred.resolve(newData);
});
});
return deferred.promise();
};
// Make initial method call
myAsyncMethod(myInitialData).then(
function(transformedData){
// transformed data from server is returned here.
}
);

Asserting values in node.js

I have a function,
Edit1 - Updated function with real one because the previous one was simplified synchronous function and the code would have worked as correctly pointed by #AlexMA in the comments
'returnSuccessOrFailure': function () {
return driver.findElement(wd.By.css('div#button')).then(function (button) {
return button.getAttribute('class').then(function (status) {
return status;
});
});
}
In my node.js test, my assertion is failing because the assert is called before returnSuccessOrFailure finishes execution.
var value = returnSuccessOrFailure();
assert.equal(value,'success', 'Looks like something failed');
If I implement a promise in returnSuccessOrFailure and chain my assert then that works. My question is do I have to implement promises all the time for such situations to block the execution? I am new to Javascript and the async nature of it and any insight when to use promises and when not to would be useful.
you don't have to "implement a promise" in, just return the one you already have:
returnSuccessOrFailure': function () {
return driver.findElement(wd.By.css('div#button')).then(function (button) {
...
but then, yes, you do still need to put your assert in a done handler
returnSuccessOrFailure().done(function(value) {
assert.equal(value,'success', 'Looks like something failed');
}
Chaining you asserts will not only make it work but will also make for more readable code. Knowing what happens in what order can be useful when going back to refactor. Not only that but the structure of callbacks/promises also allow for easily written timer tests.
Also, since your test needs to have the current state of execution, it is more than likely that writing tests with asserts in callbacks is what you will need anyway.
My question is do I have to implement promises all the time for such situations to block the execution?
Notice that promises don't block the execution. They defer the execution of code that depends on the result, notice that you're still chaining callbacks on them.
I am new to Javascript and the async nature of it and any insight when to use promises and when not to would be useful.
Promises are useful wherever you have some code that might run asynchronously and needs to pass back an asynchronous result. Otherwise you would need to use callbacks, which are way more ugly than promises.
This is part of code contracts and representing preconditions (what holds before you execute), postconditions (what holds after you execute), and object invariants (what can not change). JavaScript does not have native support for this, but you can use third party libraries (Cerny.js, ecmaDebug, jsContract, or jscategory)
I think it depends on your coding style, is it EAFP(Easier to ask for forgiveness than permission) or LBYL(Look before you leap). Both are viable! In most compiled languages you would use LBYL. However in Python for example you would use EAFP.
Generally if you know you will fail you want to fail fast. If you like to use assertions to ensure code fails fast it is up to you.

Synchronous promise resolution (bluebird vs. jQuery)

I have developed a small lib for the Dynamics CRM REST/ODATA webservice (CrmRestKit). The lib dependes on jQuery and utilizes the promise-pattern, repectivly the promise-like-pattern of jQuery.
Now I like to port this lib to bluebird and remove the jQuery dependency. But I am facing a problem because bluebird does not support the synchronous resolution of promise-objects.
Some context information:
The API of the CrmRestKit excepts an optional parameter that defines if the web-service call should be performed in sync or async mode:
CrmRestKit.Create( 'Account', { Name: "foobar" }, false ).then( function ( data ) {
....
} );
When you pass "true" or omit the last parameter, will the method created the record in sync. mode.
Sometimes it is necessary to perform a operation in sync-mode, for instance you can write JavaScript code for Dynamics CRM that is involed for the save-event of an form and in this event-handler you need to perform sync-operation for validation (e.g. validate that a certain number of child-records exist, in case the right number of records exist, cancel the save-operation and show an error message).
My problem now is the following: bluebird does not support the resolution in sync-mode. For instance when I do the following, the "then" handler is invoked in async fashion:
function print( text ){
console.log( 'print -> %s', text );
return text;
}
///
/// 'Promise.cast' cast the given value to a trusted promise.
///
function getSomeTextSimpleCast( opt_text ){
var text = opt_text || 'Some fancy text-value';
return Promise.cast( text );
}
getSomeTextSimpleCast('first').then(print);
print('second');
The output is the following:
print -> second
print -> first
I would expect that the "second" appears after the "first" because the promise is already resolved with an value. So I would assume that an then-event-handler is immediately invoked when applied on an already resolved promise-object.
When I do the same (use then on an already resolved promise) with jQuery I will have my expected result:
function jQueryResolved( opt_text ){
var text = opt_text || 'jQuery-Test Value',
dfd = new $.Deferred();
dfd.resolve(text);
// return an already resolved promise
return dfd.promise();
}
jQueryResolved('third').then(print);
print('fourth');
This will generate the following output:
print -> third
print -> fourth
Is there a way to make bluebird work in the same fashion?
Update:
The provided code was just to illustrate the problem. The idea of the lib is: Regardless of the execution-mode (sync, async) the caller will always deal with an promise-object.
Regarding "... asking the user... doesn't seems to make any sense": When you provide two methods "CreateAsync" and "CreateSync" it is also up to the user to decide how the operation is executed.
Anyway with the current implementation the default behavior (last parameter is optional) is a async execution. So 99% of the code requires a promise-object, the optional parameter is only use for the 1% cases where you simply need a sync execution. Furthermore I developed to lib for myself and I use in 99,9999% of the case the async mode but I thought it is nice to have the option to go the sync-road as you like.
But I thinks I got the point an sync method should simply return the value. For the next release (3.0) I will implement "CreateSync" and "CreateAsync".
Thanks for your input.
Update-2
My intension for the optional parameter was to ensure a consistend behavior AND prevent logic error. Assume your as a consumer of my methode "GetCurrentUserRoles" that uses lib. So the method will alway return an promise, that means you have to use the "then" method to execute code that depends on the result. So when some writes code like this, I agree it is totally wrong:
var currentUserRoels = null;
GetCurrentUserRoles().then(function(roles){
currentUserRoels = roles;
});
if( currentUserRoels.indexOf('foobar') === -1 ){
// ...
}
I agree that this code will break when the method "GetCurrentUserRoles" changes from sync to async.
But I understand that this I not a good design, because the consumer should now that he deals with an async method.
Short version: I get why you want to do that, but the answer is no.
I think the underlying question being asked is whether a completed promise should immediately run a callback, if the promise has already completed. I can think of a lot of reasons that this might happen - for example, an asynchronous save procedure that only saves data if changes were made. It may be able to detect changes from the client side in a synchronous fashion without having to go through an external resource, but if changes are detected then and only then would an asynchronous operation be required.
In other environments that have asynchronous calls, the pattern seems to be that the developer is responsible for understanding that their work might complete immediately (for example, .NET framework's implementation of the async pattern accomodates this). This is not a design problem of the framework, it's the way it's implemented.
JavaScript's developers (and many of the commenters above) seem to have a different point of view on this, insisting that if something might be asynchronous, it must always be asynchronous. Whether this is "right" or not is immaterial - according to the specification I found at https://promisesaplus.com/, item 2.2.4 states that basically no callbacks can be called until you are out of what I'll refer to as "script code" or "user code"; that is, the specification says clearly that even if the promise is completed you can't invoke the callback immediately. I've checked a few other places and they either say nothing on the topic or agree with the original source. I don't know if https://promisesaplus.com/ could be considered a definitive source of information in this regard, but no other sources that I saw disagreed with it and it seems to be the most complete.
This limitation is somewhat arbitrary and I frankly prefer the .NET perspective on this one. I'll leave it up to others to decide if they consider it "bad code" to do something that might or might not be synchronous in a way that looks asynchronous.
Your actual question is whether or not Bluebird can be configured to do the non-JavaScript behavior. Performance-wise there may be a minor benefit to doing so, and in JavaScript anything's possible if you try hard enough, but as the Promise object becomes more ubiquitous across platforms you will see a shift to using it as a native component instead of custom written polyfills or libraries. As such, whatever the answer is today, reworking a promise in Bluebird is likely to cause you problems in the future, and your code should probably not be written to depend on or provide immediate resolution of a promise.
You might think this is a problem, because there's no way to have
getSomeText('first').then(print);
print('second');
and to have getSomeText "first" printed before "second" when the resolution is synchronous.
But I think you have a logic problem.
If your getSomeText function may be synchronous or asynchronous, depending on the context, then it shouldn't impact the order of execution. You use promises to ensure it's always the same. Having a variable order of execution would likely become a bug in your application.
Use
getSomeText('first') // may be synchronous using cast or asynchronous with ajax
.then(print)
.then(function(){ print('second') });
In both cases (synchronous with cast or asynchronous resolution), you'll have the correct execution order.
Note that having a function being sometimes synchronous and sometimes not isn't a weird or unlikely case (think about cache handling, or pooling). You just have to suppose it's asynchronous, and all will be always fine.
But asking the user of the API to precise with a boolean argument if he wants the operation to be asynchronous doesn't seem to make any sense if you don't leave the realm of JavaScript (i.e. if you don't use some native code).
The point of promises is to make asynchronous code easier, i.e. closer to what you feel when using synchronous code.
You're using synchronous code. Don't make it more complicated.
function print( text ){
console.log( 'print -> %s', text );
return text;
}
function getSomeTextSimpleCast( opt_text ){
var text = opt_text || 'Some fancy text-value';
return text;
}
print(getSomeTextSimpleCast('first'));
print('second');
And that should be the end of it.
If you want to keep the same asynchronous interface even though your code is synchronous, then you have to do it all the way.
getSomeTextSimpleCast('first')
.then(print)
.then(function() { print('second'); });
then gets your code out of the normal execution flow, because it's supposed to be asynchronous. Bluebird does it the right way there. A simple explanation of what it does:
function then(fn) {
setTimeout(fn, 0);
}
Note that bluebird doesn't really do that, it's just to give you a simple example.
Try it!
then(function() {
console.log('first');
});
console.log('second');
This will output the following:
second
first
There are some good answers here already, but to sum up the crux of the matter very succinctly:
Having a promise (or other async API) that is sometimes asynchronous and sometimes synchronous is a bad thing.
You may think it's fine because the initial call to your API takes a boolean to switch off between sync/async. But what if that's buried in some wrapper code and the person using that code doesn't know about these shenanigans? They've just wound up with some unpreditable behavior through no fault of their own.
The bottom line: Don't try to do this. If you want synchronous behavior, don't return a promise.
With that, I'll leave you with this quotation from You Don't Know JS:
Another trust issue is being called "too early." In application-specific terms, this may actually involve being called before some critical task is complete. But more generally, the problem is evident in utilities that can either invoke the callback you provide now (synchronously), or later (asynchronously).
This nondeterminism around the sync-or-async behavior is almost always going to lead to very difficult to track down bugs. In some circles, the fictional insanity-inducing monster named Zalgo is used to describe the sync/async nightmares. "Don't release Zalgo!" is a common cry, and it leads to very sound advice: always invoke callbacks asynchronously, even if that's "right away" on the next turn of the event loop, so that all callbacks are predictably async.
Note: For more information on Zalgo, see Oren Golan's "Don't Release Zalgo!" (https://github.com/oren/oren.github.io/blob/master/posts/zalgo.md) and Isaac Z. Schlueter's "Designing APIs for Asynchrony" (http://blog.izs.me/post/59142742143/designing-apis-for-asynchrony).
Consider:
function result(data) {
console.log( a );
}
var a = 0;
ajax( "..pre-cached-url..", result );
a++;`
Will this code print 0 (sync callback invocation) or 1 (async callback invocation)? Depends... on the conditions.
You can see just how quickly the unpredictability of Zalgo can threaten any JS program. So the silly-sounding "never release Zalgo" is actually incredibly common and solid advice. Always be asyncing.
What about this case, also CrmFetchKit related which in latest version uses Bluebird. I have upgraded from version 1.9 that was based on jQuery. Still the old app code that uses CrmFetchKit has methods the prototypes of which I can't or won't change.
Existing App Code
CrmFetchKit.FetchWithPaginationSortingFiltering(query.join('')).then(
function (results, totalRecordCount) {
queryResult = results;
opportunities.TotalRecords = totalRecordCount;
done();
},
function err(e) {
done.fail(e);
}
);
Old CrmFetchKit implementation (a custom version of fetch())
function fetchWithPaginationSortingFiltering(fetchxml) {
var performanceIndicator_StartTime = new Date();
var dfd = $.Deferred();
fetchMore(fetchxml, true)
.then(function (result) {
LogTimeIfNeeded(performanceIndicator_StartTime, fetchxml);
dfd.resolve(result.entities, result.totalRecordCount);
})
.fail(dfd.reject);
return dfd.promise();
}
New CrmFetchKit implementation
function fetch(fetchxml) {
return fetchMore(fetchxml).then(function (result) {
return result.entities;
});
}
My problem is that the old version had the dfd.resolve(...) where I was able to pass any number of params that I need.
The new implementation just returns, the parent seems to call the callback, I can't call it directly.
I went and made a custom version of the fetch() in the new implementation
function fetchWithPaginationSortingFiltering(fetchxml) {
var thePromise = fetchMore(fetchxml).then(function (result) {
thePromise._fulfillmentHandler0(result.entities, result.totalRecordCount);
return thePromise.cancel();
//thePromise.throw();
});
return thePromise;
}
But the problem is that the callback gets called two times, once when I do it explicitly and second by the framework but it passes it one parameter only. To trick it and "tell" not to call anything because I do it explicitly I try to call .cancel() but it is ignored. I understood why but still how do you do the "dfd.resolve(result.entities, result.totalRecordCount);" in the new version with out having to changes prototypes in the app that uses this library ?
You can in fact do this, yes.
Modify the bluebird.js file (for npm: node_modules/bluebird/js/release/bluebird.js), with the following change:
[...]
target._attachExtraTrace(value);
handler = didReject;
}
- async.invoke(settler, target, {
+ settler.call(target, {
handler: domain === null ? handler
: (typeof handler === "function" &&
[...]
For more info, see here: https://github.com/stacktracejs/stacktrace.js/issues/188

Need Help Thinking How to Program Asynchronously

I'm using NodeJS to walk over a list of files and generate an MD5 hash for each one. Here's how I would normally do this synchronously:
// Assume files is already populated with an array of file objects
for(file in files) {
var currentFile = files[file];
currentFile.md5 = md5(file.path);
}
The problem here is that the MD5 function is asynchronous and actually has a callback function that is runs once the MD5 hash has been generated for the file. Thus, all of my currentFile.md5 variables are just going to be set to undefined.
Once I have gotten all of the MD5 hashes for all of the files I'll need to move onto another function to deal with that information.
How gnarly is the code going to get in order for me to do this asynchronously? What's the cleanest way to accomplish what I want to do? Are there common different approaches that I should be aware of?
To call an async function multiple times, you should make a function and call it in recursion like this.
I have assumed your md5 function has a callback with two params err and result.
var keys = Object.keys(files); // taking all keys in an array.
function fn() {
var currentFile = files[keys.shift()];
md5(currentFile, function (err, result) {
// Use result, store somewhere
// check if more files
if (keys.length) {
fn();
} else {
// done
}
});
}
One great approach is to use async. (Search on npm)
If you want to roll your own
Count the files, put that in a var
Everytime fs opens a file and calls your intermediate callback, compute and store the MD5
Also, decrement that counter.
When counter === 0, call a "final" callback, passing back all the MD5s.
To answer your questions (theoretically), in Javascript world, there are (at the moment) 2 different ways to deal with asynchronous code
Using callbacks. This is the most basic way that people start using Javascript know. However , there are plenty of libraries to help people deal with callback in a less painful way such as async, step. In your particular problem. Assuming that md5 is somehow weirdly asynchronous, you can use https://github.com/caolan/async#parallel to achieve it
Another way is to use promise, there are also plenty of promise-compliant libraries such as q, when. Basically, with a promise you have a nicer way to organize your code flow (IMO). With the problem above you can use when.all to gather the result of md5. However, you need to turn md5 into a promise-compliant function
To avoid "callback hell" you should introduce the world of promises to your Node toolset. I suggest q https://npmjs.org/package/q
Here is a post on SO that can help and give you an idea of the syntax how to use q.js promises to work with multiple asynchronous operations.
You essentially would run all your async functions with defered promises, the .then() chained method would fire when all promises are resolved and the function passed inside then() can process your MD5'd data.
I hope this helps.

Categories