I have used kriskowal's Q library for a project (web scraper / human-activity simulator) and have become acquainted with promises, returning them and resolving/rejecting them, and the library's basic asynchronous control flow methods and error-throwing/catching mechanisms have proven essential.
I have encountered some issues though. My promise.then calls and my callbacks have the uncanny tendency to form pyramids. Sometimes it's for scoping reasons, other times it's to guarantee a certain order of events. (I suppose I might be able to fix some of these problems by refactoring, but going forward I want to avoid "callback hell" altogether.)
Also, debugging is very frustrating. I spend a lot of time console.log-ing my way to the source of errors and bugs; after I finally find them I will start throwing errors there and catching them somewhere else with promise.finally, but the process of locating the errors in the first place is arduous.
Also, in my project, order matters. I need to do pretty much everything sequentially. Oftentimes I find myself generating arrays of functions that return promises and then chaining them to each other using Array.prototype.reduce, which I don't think I should have to do.
Here is an example of one of my methods that uses this reduction technique:
removeItem: function (itemId) {
var removeRegexp = new RegExp('\\/stock\\.php\\?remove=' + itemId);
return this.getPage('/stock.php')
.then(function (webpage) {
var
pageCount = 5,
promiseFunctions = [],
promiseSequence;
// Create an array of promise-yielding functions that can run sequentially.
_.times(pageCount, function (i) {
var promiseFunction = function () {
var
promise,
path;
if (i === 0) {
promise = Q(webpage);
} else {
path = '/stock.php?p=' + i;
promise = this.getPage(path);
}
return promise.then(function (webpage) {
var
removeMatch = webpage.match(removeRegexp),
removePath;
if (removeMatch !== null) {
removePath = removeitemMatch[0];
return this.getPage(removePath)
.delay(1000)
// Stop calling subsequent promises.
.thenResolve(true);
}
// Don't stop calling subsequent promises.
return false;
}.bind(this));
}.bind(this);
promiseFunctions.push(promiseFunction);
}, this);
// Resolve the promises sequentially but stop early if the item is found.
promiseSequence = promiseFunctions.reduce(function (soFar, promiseFunction, index) {
return soFar.then(function (stop) {
if (stop) {
return true;
} else {
return Q.delay(1000).then(promiseFunction);
}
});
}, Q());
return promiseSequence;
}.bind(this))
.fail(function (onRejected) {
console.log(onRejected);
});
},
I have other methods that do basically the same thing but which are suffering from much worse indentation woes.
I'm considering refactoring my project using coalan's async library. It seems similar to Q, but I want to know exactly how they differ. The impression I am getting is that async more "callback-centric" while Q is "promise-centric".
Question: Given my problems and project requirements, what would I gain and/or lose by using async over Q? How do the libraries compare? (Particularly in terms of executing series of tasks sequentially and debugging/error-handling?)
Both libraries are good. I have discovered that they serve separate purposes and can be used in tandem.
Q provides the developer with promise objects, which are future representations of values. Useful for time travelling.
Async provides the developer with asynchronous versions of control structures and aggregate operations.
An example from one attempt at a linter implementation demonstrates a potential unity among libraries:
function lint(files, callback) {
// Function which returns a promise.
var getMerged = merger('.jslintrc'),
// Result objects to invoke callback with.
results = [];
async.each(files, function (file, callback) {
fs.exists(file, function (exists) {
// Future representation of the file's contents.
var contentsPromise,
// Future representation of JSLINT options from .jslintrc files.
optionPromise;
if (!exists) {
callback();
return;
}
contentsPromise = q.nfcall(fs.readFile, file, 'utf8');
optionPromise = getMerged(path.dirname(file));
// Parallelize IO operations.
q.all([contentsPromise, optionPromise])
.spread(function (contents, option) {
var success = JSLINT(contents, option),
errors,
fileResults;
if (!success) {
errors = JSLINT.data().errors;
fileResults = errors.reduce(function (soFar, error) {
if (error === null) {
return soFar;
}
return soFar.concat({
file: file,
error: error
});
}, []);
results = results.concat(fileResults);
}
process.nextTick(callback);
})
.catch(function (error) {
process.nextTick(function () {
callback(error);
});
})
.done();
});
}, function (error) {
results = results.sort(function (a, b) {
return a.file.charCodeAt(0) - b.file.charCodeAt(0);
});
callback(error, results);
});
}
I want to do something potentially-blocking for each file. So async.each is the obvious choice. I can parallelize related operations per-iteration with q.all and reuse my option values if they apply to 2 or more files.
Here, Async and Q each influence the control flow of the program, and Q represents values resolving to file contents sometime in the future. The libraries work well together. One does not need to "choose one over the other".
Callback pyramids in your code can be simplified using promise composition and javascript lexical scoping.
removeItem: function (itemId) {
var removeRegexp = new RegExp('\\/stock\\.php\\?remove=' + itemId);
var found = false
var promise = getPage('/sock.php')
_.times(5, (i) => {
promise = promise.then((webpage) => {
if (found) return true
var removeMatch = webpage.match(removeRegexp)
var found = removeMath !== null
var nextPage = found ? removeMatch[0] : '/stock.php?p='+i+1
return Q.delay(1000).then(() => this.getPage(nextPage))
})
})
return promise.fail(console.log.bind(console))
},
IMHO async should not be used in new javascript code. Promises are more composable, and allow for a lot more intutive code.
The primary reason why node did not use promises was because of performance concerns which have largely been addressed very well by libraries like Bluebird and Q.
As async/await syntax becomes more mainstream, promises will pave the way for code that looks very similar with synchronous code.
While this is still not an actual answer to my question (Q vs async), regarding my problem, I've found Selenium / WebDriverJs to be a viable solution.
driver.get('http://www.google.com');
driver.findElement(webdriver.By.name('q')).sendKeys('webdriver');
driver.findElement(webdriver.By.name('btnG')).click();
driver.wait(function() {
return driver.getTitle().then(function(title) {
return title === 'webdriver - Google Search';
});
}, 1000);
WebDriver uses a queue to execute promises sequentially, which helps immensely with controlling indentation. Its promises are also compatible with Q's.
Creating a sequence of promises is no longer an issue. A simple for-loop will do.
As for stopping early in a sequence, don't do this. Instead of using a sequence, use an asynchronous-while design and branch.
Related
This question already has answers here:
Is it useful to always return a promise
(2 answers)
When to make a function deferred using Promises
(2 answers)
Closed 5 years ago.
I am currently trying to beautify some NodeJS legacy code and doing so came up with this questions where I could not find any best-practice or go-to-guide (if you know any, please share it with me).
The aim is to not simply see that one can combine synchronous and asynchronous code via Promises (this is clear), but why one should e.g. go for synchronous code if the synchronous processing of a function is clear. This means what disadvantages would it have to simply add a Promise to the synchronous code? Are there any? E.g. in the runtime or, more important, in the asynchronous process flow?
E.g. take the following example for an async function:
Foo.prototype.addNode = function addNode() {
var self = this;
return new Promise(function (resolve, reject) {
var node = new Node(self.getNodes().getLastId() + 1);
resolve (self.getNodes().add(node));
});
}
In our business logic this function should indeed be async, since this is called recursively in a promise chain.
However, the function add is handled synchronously, since this is a simple Array.push() of a generic element with some validation.
GenericElementList.prototype.add = function add(T) {
if (this.T.length === 0) {
this.T.push(T);
this.typeOfT = T.constructor;
return this;
}
else {
if (!(T.constructor === this.typeOfT)){
this.T.push(T);
return this;
}
else {
throw new IllegalArgumentError('Invalid type of element to push into array! Was: ' + T.constructor.name + ' but should be: ' + this.typeOfT.name + '!');
}
}
};
(So to avoid confusion: self.getNodes() returns an element Node which is a GenericElement and the GenericElementList is a list data structure for GenericElements.)
Now my question is: Would there be any disadvantages regarding runtime or process flow in the asynchronous processing of these methods, if we would add a Promise to the add-function? Are there any reasons to avoid that? Or would the only disadvantage be some boilerplate code?
In javascript/node sync execution remain sync no matter how do you express it. take an example
console.log("stage 1")
findIndex(data, 2, (err, index)=>{
console.log(err, index)
})
console.log("stage 3")
function findIndex (arr, val, callback){
console.log("stage 2")
var idx =arr.indexOf(val)
if(idx==-1) return callback(new Error("value not found"))
return callback(null, idx)
}
while async or can say I/O bound execution always be async. No matter how do you express it (some people thought es6 async/await makes async execution to sync)
console.log("stage 1")
setTimeout(()=>{
console.log("stage 2")
}, 0)
console.log("stage 3")
Open your js console and find difference and flow of execution
Promise also follow same thing
Now async function inside loop
sequential iteration useful when async tasks are dependent. (result of one operation needs feed other(/s) )
(function loop(i) {
if(i==4) return
setTimeout(function() {
console.log('Working on index', i);
loop(++i)
}, 1000);
})(0);
parallel execution is useful when async tasks are independent to each other.
var data = [1,2,3,4]
data.map(val=>{
return new Promise((resolve, reject) => {
setTimeout(function() {
console.log('Working on index', val);
// pass data in reslove() and get when promise resloved
})
});
})
Promise.all(data)
.then(done=>{
// get passed data
console.log(done);
})
.catch(err=>{
console.log(err);
})
get more from
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise
I have many asynchronous methods to execute and my program flows can change a lot depending on each method return. The logic below is one example. I could not write it in a easy-to-read way using Promises. How would you write it?
Ps: more complex flows are welcome.
Ps2: is_business is a predefined flag where we say whether we are writing a "business user" or a "person user".
begin transaction
update users
if updated
if is_business
update_business
if not updated
insert business
end if
else
delete business
end if
else
if upsert
insert user
if is_business
insert business
end if
end if
end if
commit transaction
The nice thing about promises is that they make a simple analogy between synchronous code and asynchronous code. To illustrate (using the Q library):
Synchronous:
var thisReturnsAValue = function() {
var result = mySynchronousFunction();
if(result) {
return getOneValue();
} else {
return getAnotherValue();
}
};
try {
var value = thisReturnsAValue();
console.log(value);
} catch(err) {
console.error(err);
}
Asynchronous:
var Q = require('q');
var thisReturnsAPromiseForAValue = function() {
return Q.Promise(function() {
return myAsynchronousFunction().then(function(result) {
if(result) {
// Even getOneValue() would work here, because a non-promise
// value is automatically cast to a pre-resolved promise
return getOneValueAsynchronously();
} else {
return getAnotherValueAsynchronously();
}
});
});
};
thisReturnsAPromiseForAValue().then(function(value) {
console.log(value);
}, function(err) {
console.error(err);
});
You just need to get used to the idea that return values are always accessed as arguments to then-callbacks, and that chaining promises equates to composing function calls (f(g(h(x)))) or otherwise executing functions in sequence (var x2 = h(x); var x3 = g(x2);). That's essentially it! Things get a little tricky when you introduce branches, but you can figure out what to do from these first principles. Because then-callbacks accept promises as return values, you can mutate a value you got asynchronously by returning another promise for an asynchronous operation which resolves to a new value based on the old one, and the parent promise will not resolve until the new one resolves! And, of course, you can return these promises from within if-else branches.
The other really nice thing illustrated in the example above is that promises (at least ones that are compliant with Promises/A+) handle exceptions in an equally analogous way. The first error "raised" bypasses the non-error callbacks and bubbles up to the first available error callback, much like a try-catch block.
For what it's worth, I think trying to mimic this behavior using hand-crafted Node.js-style callbacks and the async library is its own special kind of hell :).
Following these guidelines your code would become (assuming all functions are async and return promises):
beginTransaction().then(function() {
// beginTransaction() has run
return updateUsers(); // resolves the boolean value `updated`
}).then(function(updated) {
// updateUsers() has "returned" `updated`
if(updated) {
if(isBusiness) {
return updateBusiness().then(function(updated) {
if(!updated) {
return insertBusiness();
}
// It's okay if we don't return anything -- it will
// result in a promise which immediately resolves to
// `undefined`, which is a no-op, just like a missing
// else-branch
});
} else {
return deleteBusiness();
}
} else {
if(upsert) {
return insertUser().then(function() {
if(isBusiness) {
return insertBusiness();
}
});
}
}
}).then(function() {
return commitTransaction();
}).done(function() {
console.log('all done!');
}, function(err) {
console.error(err);
});
The solution is a mix of #mooiamaduck answer and #Kevin comment.
Using promises, ES6 generators and co library makes the code much clearer. I found a good example when reading a postgresql node library example (pg). In the example below pool.connect and client.query are asynchronous operations that returns Promises. We can easily add an if/else after geting result and then make more async operations keeping code looking like synchronous.
co(function * () {
var client = yield pool.connect()
try {
yield client.query('BEGIN')
var result = yield client.query('SELECT $1::text as name', ['foo'])
yield client.query('INSERT INTO something(name) VALUES($1)', [result.rows[0].name])
yield client.query('COMMIT')
client.release()
} catch(e) {
// pass truthy value to release to destroy the client
// instead of returning it to the pool
// the pool will create a new client next time
// this will also roll back the transaction within postgres
client.release(true)
}
})
I'm trying the es6-promisify module in nodejs in a simple example similiar to the example on the npm page of es6-promisify.
But I want to use a forEach loop to check the stats of the entries and push them to a separate array if they are directories:
var fs=require('fs');
var promisify=require('es6-promisify');
var path='../test_folder/';
function listSubfolderNamesSync(fpath){
var arrayOfFolders=[];
var array=fs.readdirSync(fpath);
array.forEach(function(element,index,array){
var stat=promisify(fs.stat);
stat(path+element).then(function(stats){
if (stats.isDirectory()){
arrayOfFolders.push(element);
console.log('pushed');
}
})
});
return arrayOfFolders;
}
console.log(listSubfolderNamesSync(path));
With 2 subfolders, the output would be:
[]
pushed
pushed
I know why my code doesn't work like I want, but I can't figure out how to solve this problem elegantly. I have seen code samples with Promise.all() and the usage of Array.map() to have an array of promises and wait for all of them to resolve before go forward in the chain. The problem I see is that I can't/don't want to create a promise explicitly for each time I check for the stats and put those promises in an array to use Promise.all(array). I have the feeling that the solution would require quite a work-around and would greatly reduce code readability.
Maybe it's better just to use this in the forEach-loop, which is my working code before I wanted to try to promisify:
if(fs.statSync(path+element).isDirectory()){
arrayOfFolders.push(element);
}
Probably it is due to my lack of programming experience, but it seems like mixing promises, callbacks and synchronous code is not as simple as it is written in books?
As stat() returns a promise, so must your listSubfolderNamesSync() function, making it listSubfolderNamesAsync(). Unless a synchronous alternative to stat() is available, this is unavoidable.
Inside listSubfolderNamesAsync(), you need to map the array returned by fs.readdirSync() to an array of promises, then use Promise.all() to aggregatge those promises (and the data they deliver) into a single promise.
var fs = require('fs');
var promisify = require('es6-promisify');
var stat = promisify(fs.stat);
var path = '../test_folder/';
function listSubfolderNamesAsync(fpath) {
var promises = fs.readdirSync(fpath).map(function(element) {
return stat(path + element).then(function(stats) {
return stats.isDirectory() ? element : null;
});
});
return Promise.all(promises).then(function(results) {
return results.filter(function(res) {
return res !== null;
});
});
}
Now handle the results with .then() :
listSubfolderNamesAsync(path).then(function(folders) {
console.log(folders);
});
There's no great performance penalty in having a more general function that returns separate lists of folders and files. You could write something like this :
function listSubElementsAsync(fpath) {
var promises = fs.readdirSync(fpath).map(function(element) {
return stat(path + element).then(function(stats) {
var type = stats.isDirectory() ? 'folder' : 'file';
return {type:type, element:element};
});
});
return Promise.all(promises).then(function(results) {
var files = results.filter(obj => obj.type === 'file').map(obj => obj.element);
var folders = results.filter(obj => obj.type === 'folder').map(obj => obj.element);
return {files:files, folders:folders};
});
}
listSubElementsAsync(path).then(function(list) {
console.log(list.files);
console.log(list.folders);
});
I have a chain of promises that looks like this:
module.exports.deleteCommunityFollower = function deleteCommunityFollower(req, res){
var communityId = req.params.userId;
var followerId = req.session.passport.user.userId;
var community = new user_model.User(communityId);
community.getFollower(followerId)
.then(function(data) {
if(data.length === 0) {
res.sendStatus(404); //no follower found, interrupt execution
} else {
return community.removeFollower(data[0]); //returns a promise
}
})
.then(function() {
res.sendStatus(201); //follower removed, success
})
.fail(function(error) {
errorHelper.diagnosticsUploader(error, community);
res.sendStatus(500);
});
}
My question here is about line res.sendStatus(404). Is this a correct and elegant way of interrupting execution of a chain of promises? The background is, sometimes when chaining promises, I've found scenarios like this one, where you need to stop the execution of the chain for reasons that are not an error. I know I could throw an artificial error upon data.length === 0, but that just looks inelegant to me.
In the code above, when data.length === 0 is true, I simply return an http response and do not return any value to the promise resolver, thus effectively preventing the chain execution to continue. However, I'd like to validate if this is recommended practice. Leaving a promise hanging mid-way looks to me like it can be a source of trouble in the future (memory leaks?)
Since you are using modern node, here is how I would write it using Q.async:
const deleteFollower = Q.async(function*(communityId, followerId){
const community = new user_model.User(communityId);
let followers = yield community.getFollower(followerId);
if(followers.length) === 0; return false;
yield community.removeFollower(follower[0]);
return true;
});
Reads like a synchronous function and completely flat, nice huh?
I omitted the code extracting things from req/res since that would make the code harder to test and it should probably be separated anyway. I'd call it like:
function handler(req, res){
var communityId = req.params.userId;
var followerId = req.session.passport.user.userId;
deleteFollower(communityId, followerId).then(val => {
if(val) res.sendStatus(201);
else res.sendStatus(404);
}).fail(err => {
res.sendStatus(500);
errorHelper.diagnosticsUploader(err);
});
}
(Note, personally I much prefer using the bluebird library for performance reasons, where I'd use Promise.coroutine).
Given this code which counts the files in the given directory and subdirectories, how do I go about indicating that the operation is done?
function walkDirs(dirPath) {
var fs = require('fs'),
path = require('path'),
events = require('events'),
count = 0,
emitter = new events.EventEmitter();
function walkDir(dirPath) {
function readDirCallback(err, entries) {
for (var idx in entries) {
var fullPath = path.join(dirPath, entries[idx]);
(function statHandler(fullPath) {
fs.stat(fullPath, function statEach(err, stats) {
if (stats) {
if (stats.isDirectory()) {
walkDir(fullPath);
} else if (stats.isFile()) {
count += 1;
emitter.emit('counted', count, fullPath);
}
}
});
})(fullPath);
}
}
fs.readdir(dirPath, readDirCallback);
}
walkDir(dirPath);
return emitter;
}
var walker = walkDirs('C:');
I've tried specifically,
firing an event to indicate "doneness" at a place I thought appropriate, specifically after fs.readdir(dirPath, readDirCallback) call.
modifying statHandler() to return the count added. (I realized that this is effectively no different from incrementing count inside that function.
Both of these failed, because when checked, the value of count is 0. I've determined that I'm not waiting until the operation (counting the files) is done. Obviously, I need to fire a callback or event when done to get the right count.
I know the code is successfully counting, because when attaching a debugger, the count value is as expected.
At this point, I've fairly certainly determined that I have no idea how to further proceed. Specically -
How do I implement indicating "doneness" in an asynchronous operation?
Asynchronous Javascript functions generally call callback functions when they are finished. In this case, a done event would be appropriate if you have other events. Promises are now often preferred over just callbacks, but you do need to understand callbacks first.
Since readdir is asynchronous, the execution being on the next line doesn't mean its finished. That's what's confusing about asynchronous code compared to synchronous code. If I were you, I would use the debugger and step through some really simple (simpler than this) async examples. It does take awhile to get used to and is tricky.
For walking directories see https://www.npmjs.com/package/walk . You don't want to re-invent the wheel here. Always be sure to Google and/or search on npmjs for existing modules.
Once you are sure you really understand callbacks and asynchronous code then you can move on to the async module, and after that, promises with bluebird, ES6 promises, etc. Down the line something like this https://www.npmjs.com/package/co may be useful.
You can achieve this by using promises. In this example I chose Q.
npm install q
You resolve your promise whenever you consider the async function to be done by calling the .resolve() function. When your promise gets resolved it will call your success-callback in then .then() function of you promise object, which is walkDirs. The .then() function get triggered whenever your promises get resolved or rejected. If you reject you promise the error-callback will be called.
var q = require('q');
function walkDirs(dirPath) {
var deffered = q.defer();
var fs = require('fs'),
path = require('path'),
events = require('events'),
count = 0,
emitter = new events.EventEmitter();
function walkDir(dirPath) {
function readDirCallback(err, entries) {
for (var idx in entries) {
var fullPath = path.join(dirPath, entries[idx]);
(function statHandler(fullPath) {
fs.stat(fullPath, function statEach(err, stats) {
if (stats) {
if (stats.isDirectory()) {
walkDir(fullPath);
} else if (stats.isFile()) {
count += 1;
emitter.emit('counted', count, fullPath);
deffered.resolve(emitter); // resolve promise
}
}
});
})(fullPath);
}
}
fs.readdir(dirPath, readDirCallback);
}
walkDir(dirPath);
return q.promise;
}
walkDirs('C:')
.then(success, error) //can also take error callback if promise is rejected.
function success(function(data) {
//data = resolved data.
console.log("is successfully done");
})
function errer(function() {
console.log("is errorly done");
})