We Keep Coding

Avoiding javascript callback and promise hell

I have many asynchronous methods to execute and my program flows can change a lot depending on each method return. The logic below is one example. I could not write it in a easy-to-read way using Promises. How would you write it?
Ps: more complex flows are welcome.
Ps2: is_business is a predefined flag where we say whether we are writing a "business user" or a "person user".
begin transaction
update users
if updated
if is_business
update_business
if not updated
insert business
end if
else
delete business
end if
else
if upsert
insert user
if is_business
insert business
end if
end if
end if
commit transaction

The nice thing about promises is that they make a simple analogy between synchronous code and asynchronous code. To illustrate (using the Q library):
Synchronous:
var thisReturnsAValue = function() {
var result = mySynchronousFunction();
if(result) {
return getOneValue();
} else {
return getAnotherValue();
}
};
try {
var value = thisReturnsAValue();
console.log(value);
} catch(err) {
console.error(err);
}
Asynchronous:
var Q = require('q');
var thisReturnsAPromiseForAValue = function() {
return Q.Promise(function() {
return myAsynchronousFunction().then(function(result) {
if(result) {
// Even getOneValue() would work here, because a non-promise
// value is automatically cast to a pre-resolved promise
return getOneValueAsynchronously();
} else {
return getAnotherValueAsynchronously();
}
});
});
};
thisReturnsAPromiseForAValue().then(function(value) {
console.log(value);
}, function(err) {
console.error(err);
});
You just need to get used to the idea that return values are always accessed as arguments to then-callbacks, and that chaining promises equates to composing function calls (f(g(h(x)))) or otherwise executing functions in sequence (var x2 = h(x); var x3 = g(x2);). That's essentially it! Things get a little tricky when you introduce branches, but you can figure out what to do from these first principles. Because then-callbacks accept promises as return values, you can mutate a value you got asynchronously by returning another promise for an asynchronous operation which resolves to a new value based on the old one, and the parent promise will not resolve until the new one resolves! And, of course, you can return these promises from within if-else branches.
The other really nice thing illustrated in the example above is that promises (at least ones that are compliant with Promises/A+) handle exceptions in an equally analogous way. The first error "raised" bypasses the non-error callbacks and bubbles up to the first available error callback, much like a try-catch block.
For what it's worth, I think trying to mimic this behavior using hand-crafted Node.js-style callbacks and the async library is its own special kind of hell :).
Following these guidelines your code would become (assuming all functions are async and return promises):
beginTransaction().then(function() {
// beginTransaction() has run
return updateUsers(); // resolves the boolean value `updated`
}).then(function(updated) {
// updateUsers() has "returned" `updated`
if(updated) {
if(isBusiness) {
return updateBusiness().then(function(updated) {
if(!updated) {
return insertBusiness();
}
// It's okay if we don't return anything -- it will
// result in a promise which immediately resolves to
// `undefined`, which is a no-op, just like a missing
// else-branch
});
} else {
return deleteBusiness();
}
} else {
if(upsert) {
return insertUser().then(function() {
if(isBusiness) {
return insertBusiness();
}
});
}
}
}).then(function() {
return commitTransaction();
}).done(function() {
console.log('all done!');
}, function(err) {
console.error(err);
});

The solution is a mix of #mooiamaduck answer and #Kevin comment.
Using promises, ES6 generators and co library makes the code much clearer. I found a good example when reading a postgresql node library example (pg). In the example below pool.connect and client.query are asynchronous operations that returns Promises. We can easily add an if/else after geting result and then make more async operations keeping code looking like synchronous.
co(function * () {
var client = yield pool.connect()
try {
yield client.query('BEGIN')
var result = yield client.query('SELECT $1::text as name', ['foo'])
yield client.query('INSERT INTO something(name) VALUES($1)', [result.rows[0].name])
yield client.query('COMMIT')
client.release()
} catch(e) {
// pass truthy value to release to destroy the client
// instead of returning it to the pool
// the pool will create a new client next time
// this will also roll back the transaction within postgres
client.release(true)
}
})

How to promisify this Mongoose code?

I am trying to use Mongoose's built in promise support to write some clean Javascript code for a user sending a friend request to another. However, when I try to ensure proper error handling and sequentiality, I still end up with a (slightly smaller than normal) pyramid of doom.
Here, I first ensure that the friend request is valid, then save the target's Id to the requester's sent requests then, if that save was successful, save the requester's Id to the target's friend requests.
Do I need to use a third party library like q in order to do this as cleanly as possible? How can I structure this such that I can use the traditional single error handler at the end?
function _addFriend (requesterId, targetId) {
// (integer, integer)
User.findById(requesterId)
.exec((requester) => {
if (!(targetId in requester.friends
|| targetId in requester.sentfriendRequests
|| targetId in requester.friendRequests)) {
requester.sentfriendRequests = requester.sentfriendRequests.concat([targetId])
requester.save()
.then((err) => {
if (err) throw err;
User.findById(targetId)
.exec((err, target) => {
if (err) throw err;
target.friendRequests = target.friendRequests.concat([requesterId])
target.save().then(err => {if (err) throw err})
})
})
}
})
}

You will need some nesting to do conditionals in promise code, but not as much as with callback-based code.
You seem to have messed up a bit of the if (err) throw err; stuff, you should never need that with promises. Just always use .then(result => {…}), and don't pass callbacks to exec any more.
If you always properly return promises from your asynchronous functions (including then callbacks for chaining), you can add the single error handler in the end.
function _addFriend (requesterId, targetId) {
// (integer, integer)
return User.findById(requesterId).exec().then(requester => {
if (targetId in requester.friends
|| targetId in requester.sentfriendRequests
|| targetId in requester.friendRequests) {
return;
}
requester.sentfriendRequests = requester.sentfriendRequests.concat([targetId])
return requester.save().then(() => {
return User.findById(targetId).exec()
}).then(target => {
target.friendRequests = target.friendRequests.concat([requesterId])
return target.save()
});
});
}
_addFriend(…).catch(err => {
…
})

In English, the way to do this is to use the promises returned by exec() have then blocks return promises, un-indent, then add then to those. Much easier to say in code...
EDIT thanks (again) to #Bergi for making me read and understand the app logic. #Bergi is right that there must be a little nesting to get the job done, but the real point isn't about reducing nesting, but about improving clarity.
Better clarity can come from factoring into logical parts, including some that return in promises.
These few functions conceal the promise nesting that's required by the logic. This doesn't specify (because the OP doesn't indicate how the app should handle) what addFriend should return when it refuses to do so due to an existing request...
function _addFriend (requesterId, targetId) {
// note - pass no params to exec(), use it's returned promise
return User.findById(requesterId).exec().then((requester) => {
return canAddFriend(requester, targetId) ? addFriend(requester, targetId) : null;
});
}
function canAddFriend(requester, targetId) {
return requester && targetId &&
!(targetId in requester.friends
|| targetId in requester.sentfriendRequests
|| targetId in requester.friendRequests);
}
function addFriend(requester, targetId) {
requester.sentfriendRequests = requester.sentfriendRequests.concat([targetId]);
return requester.save().then(() => {
return User.findById(targetId).exec();
}).then((target) => {
target.friendRequests = target.friendRequests.concat([requesterId]);
return target.save();
});
}

Once you realise that .exec() returns a promise, you can :
achieve the desired flattening and make the code more readable.
avoid the need to handle errors amongst the "success" code.
handle errors in a terminal .then() or .catch().
As a bonus you can also (more readily) throw meaningful errors for each of those x in y conditions.
Straightforwardly, you could write :
function _addFriend(requesterId, targetId) {
return User.findById(requesterId).exec().then(requester => {
if (targetId in requester.friends) {
throw new Error('target is already a friend');
}
if (targetId in requester.sentfriendRequests) {
throw new Error('friend request already sent to target');
}
if (targetId in requester.friendRequests) {
throw new Error('target already sent a friend request to requester');
}
requester.sentfriendRequests = requester.sentfriendRequests.concat([targetId]); // or just .push()?
return requester.save();
}).then(() => {
return User.findById(targetId).exec().then(target => {
target.friendRequests = target.friendRequests.concat([requesterId]); // or just .push()?
return target.save();
});
});
}
Note the need for returns to control flow.
But you could do even better. As writtten above, the requested stuff could succeed then the target stuff fail, resulting in a db disparity. So what you really want is a db transaction to guarantee that both happen or neither. Mongoose undoubtedly provides for transactions however you can do something client-side to give you something transaction-like with partial benefit.
function _addFriend(requesterId, targetId) {
return Promise.all([User.findById(requesterId).exec(), User.findById(targetId).exec()]).then(([requester, target]) => { // note destructuring
if (targetId in requester.friends) {
throw new Error('target is already a friend');
}
if (targetId in requester.sentfriendRequests) {
throw new Error('friend request already sent to target');
}
if (targetId in requester.friendRequests) {
throw new Error('target already sent a friend request to requester');
}
requester.sentfriendRequests = requester.sentfriendRequests.concat([targetId]);
target.friendRequests = target.friendRequests.concat([requesterId]);
return requester.save().then(() => {
return target.save();
});
});
}
Here, you could still get the (unlikely) situation that the first save is successful and the second save fails, but at least you have the assurance that absolutely nothing happens unless both the requester and target exist.
In both cases, call as follows :
_addFriend(requesterId, targetId).then(function() {
// do whatever on success
}, function(error) {
// do whatever on error
});
Even if you don't use the error messages in the live environment, they could be very useful when testing/debugging. Please check them - I may have gotten them wrong.

(node:5540) Warning: a promise was created in a handler but was not returned from it

The following code:
function handleError(res, statusCode) {
statusCode = statusCode || 500;
return function(err) {
res.status(statusCode).send(err);
};
}
function respondWithResult(res, statusCode) {
statusCode = statusCode || 200;
return function(entity) {
if (entity) {
res.status(statusCode).json(entity);
}
};
}
// Creates a new Store in the DB
export function create(req, res) {
// create user
let user = req.body.user;
let store = req.body.store;
auth.hash(user.password)
.then(hash => {
user.password = hash;
// Create user, then create store, attach store object id to user, and attach user object id to store
User.create(user)
.then(userRes => {
store.owner = userRes._id;
store.memebers = [];
store.memebers.push(store.owner);
Store.create(store)
.then(storeRes => {
return respondWithResult(res, 201);
})
.catch(err => handleError(err));
})
.catch(err => handleError(err));
})
.catch(err => handleError(err));
}
prints the error mentioned in the title, "(node:5540) Warning: a promise was created in a handler but was not returned from it". I have tried changing and tweaking the code but the error still persists.

This warning is because your code is creating promises inside of .then() handlers, but not returning them from those handlers.
Change:
User.create(user)
to:
return User.create(user)
And, change:
Store.create(store)
to
return Store.create(store)
When you don't return these promises that are created inside .then() handlers, they become separate, independent promise chains and are not linked to the previous promise chain. This is usually a programming mistake which is why Bluebird makes it a warning.
When you return them, then they add to the promise chain and thus the parent promise waits for their completion before continuing on with the chain.
I'd also suggest you probably want to change:
auth.hash(user.password)
to:
return auth.hash(user.password)
So that the caller of create() can tell when everything is done.
And, you only need one .catch() handler at the highest level. Rejected promises propagate up to the top level for you automatically (one of the things that makes error handling when using promises easier).

This is a warning message when you don't return to a request. Of course this seems just another warning but when you work on a big application this becomes a very big headache because this will lead to a memory leak and wont release the memory until you restart your app or might crush your server.
you also need to return for else statments:
function respondWithResult(res, statusCode) {
statusCode = statusCode || 200;
return function(entity) {
if (entity) {
res.status(statusCode).json(entity);
}else{
//you should write an else statement also
//maybe something like this
res.status(statusCode).send(err);
}
};
}
Return to your request in every case.

Managing promise dependencies

I'm using Node.js and Bluebird to create some fairly complicated logic involving uncompressing a structured file, parsing JSON, creating and making changes to several MongoDB documents, and writing related files in multiple locations. I also have fairly complicated error handling for all of this depending on the state of the system when an error occurs.
I am having difficulty thinking of a good way to manage dependencies through the flow of promises.
My existing code basically looks like this:
var doStuff = function () {
var dependency1 = null;
var dependency2 = null;
promise1()
.then(function (value) {
dependency1 = value;
return promise2()
.then(function (value) {
dependency2 = value;
return promise3(dependency1)
.then(successFunction);
});
})
.catch(function (err) {
cleanupDependingOnSystemState(err, dependency1, dependency2);
});
};
Note that dependency1 isn't needed until promise3, and that the error handler needs to know about the dependencies.
To me this seems like spaghetti code (and my actual code is far worse with a lot of parallel control flow). I've also read that returning another promise inside of a .then callback is an antipattern. Is there a better/cleaner way of accomplishing what I'm trying to do?

I find both answers currently provided nice but clumsy. They're both good but contain overhead I don't think you need to have. If you instead use promises as proxies you get a lot of things for free.
var doStuff = function () {
var p1 = promise1();
var p2 = p1.then(promise2);
var p3 = p1.then(promise3); // if you actually need to wait for p2 here, do.
return Promise.all([p1, p2, p3]).catch(function(err){
// clean up based on err and state, can unwrap promises here
});
};
Please do not use successFunction and such it is an anti-pattern and loses information.
If you feel like you have to use successFunction you can write:
var doStuff = function () {
var p1 = promise1();
var p2 = p1.then(promise2);
var p3 = p1.then(promise3); // if you actually need to wait for p2 here, do.
Promise.join(p1, p2, p3, successFunction).catch(function(err){
// clean up based on err and state, can unwrap promises here
});
};
However, it is infinitely worse since it won't let the consumer handle errors they may be able to handle.

This question might be more appropriate for code review but here is how I'd approach it given this example:
var doStuff = function () {
// Set up your promises based on their dependencies. In your example
// promise2 does not use dependency1 so I left them unrelated.
var dep1Promise = promise1();
var dep2Promise = promise2();
var dep3Promise = dependency1Promise.then(function(value){
return promise3(value);
});
// Wait for all the promises the either succeed or error.
allResolved([dep1Promise, dep2Promise, dep3Promise])
.spread(function(dep1, dep2, dep3){
var err = dep1.error || dep2.error || dep3.error;
if (err){
// If any errored, call the function you prescribed
cleanupDependingOnSystemState(err, dep1.value, dep2.value);
} else {
// Call the success handler.
successFunction(dep3.value);
}
};
// Promise.all by default just fails on the first error, but since
// you want to pass any partial results to cleanupDependingOnSystemState,
// I added this helper.
function allResolved(promises){
return Promise.all(promises.map(function(promise){
return promise.then(function(value){
return {value: value};
}, function(err){
return {error: err};
});
});
}
The use of allResolved is only because of your callback specifics, if you had a more general error handler, you could simply resolve using Promise.all directly, or even:
var doStuff = function () {
// Set up your promises based on their dependencies. In your example
// promise2 does not use dependency1 so I left them unrelated.
var dep1Promise = promise1();
var dep2Promise = promise2();
var dep3Promise = dependency1Promise.then(function(value){
return promise3(value);
});
dep3Promise.then(successFunction, cleanupDependingOnSystemState);
};

It is certainly not an antipattern to return promises within thens, flattening nested promises is a feature of the promise spec.
Here's a possible rewrite, though I'm not sure it's cleaner:
var doStuff = function () {
promise1()
.then(function (value1) {
return promise2()
.then(function (value2) {
return promise3(value1)
.then(successFunction)
.finally(function() {
cleanup(null, value1, value2);
});
})
.finally(function() {
cleanup(null, value1, null);
});
})
.finally(function () {
cleanup(null, null, null);
});
};
Or another option, with atomic cleanup functions:
var doStuff = function () {
promise1()
.then(function (value1) {
return promise2()
.then(function (value2) {
return promise3(value1)
.then(successFunction)
.finally(function() {
cleanup3(value2);
});
})
.finally(function() {
cleanup2(value1);
});
})
.finally(function (err) {
cleanup1(err);
});
};
Really, I feel like there's not much you can do to clean this up. Event with vanilla try/catches, the best possible pattern is pretty similar to these.

Node.js Asynchronous Library Comparison - Q vs Async

I have used kriskowal's Q library for a project (web scraper / human-activity simulator) and have become acquainted with promises, returning them and resolving/rejecting them, and the library's basic asynchronous control flow methods and error-throwing/catching mechanisms have proven essential.
I have encountered some issues though. My promise.then calls and my callbacks have the uncanny tendency to form pyramids. Sometimes it's for scoping reasons, other times it's to guarantee a certain order of events. (I suppose I might be able to fix some of these problems by refactoring, but going forward I want to avoid "callback hell" altogether.)
Also, debugging is very frustrating. I spend a lot of time console.log-ing my way to the source of errors and bugs; after I finally find them I will start throwing errors there and catching them somewhere else with promise.finally, but the process of locating the errors in the first place is arduous.
Also, in my project, order matters. I need to do pretty much everything sequentially. Oftentimes I find myself generating arrays of functions that return promises and then chaining them to each other using Array.prototype.reduce, which I don't think I should have to do.
Here is an example of one of my methods that uses this reduction technique:
removeItem: function (itemId) {
var removeRegexp = new RegExp('\\/stock\\.php\\?remove=' + itemId);
return this.getPage('/stock.php')
.then(function (webpage) {
var
pageCount = 5,
promiseFunctions = [],
promiseSequence;
// Create an array of promise-yielding functions that can run sequentially.
_.times(pageCount, function (i) {
var promiseFunction = function () {
var
promise,
path;
if (i === 0) {
promise = Q(webpage);
} else {
path = '/stock.php?p=' + i;
promise = this.getPage(path);
}
return promise.then(function (webpage) {
var
removeMatch = webpage.match(removeRegexp),
removePath;
if (removeMatch !== null) {
removePath = removeitemMatch[0];
return this.getPage(removePath)
.delay(1000)
// Stop calling subsequent promises.
.thenResolve(true);
}
// Don't stop calling subsequent promises.
return false;
}.bind(this));
}.bind(this);
promiseFunctions.push(promiseFunction);
}, this);
// Resolve the promises sequentially but stop early if the item is found.
promiseSequence = promiseFunctions.reduce(function (soFar, promiseFunction, index) {
return soFar.then(function (stop) {
if (stop) {
return true;
} else {
return Q.delay(1000).then(promiseFunction);
}
});
}, Q());
return promiseSequence;
}.bind(this))
.fail(function (onRejected) {
console.log(onRejected);
});
},
I have other methods that do basically the same thing but which are suffering from much worse indentation woes.
I'm considering refactoring my project using coalan's async library. It seems similar to Q, but I want to know exactly how they differ. The impression I am getting is that async more "callback-centric" while Q is "promise-centric".
Question: Given my problems and project requirements, what would I gain and/or lose by using async over Q? How do the libraries compare? (Particularly in terms of executing series of tasks sequentially and debugging/error-handling?)

Both libraries are good. I have discovered that they serve separate purposes and can be used in tandem.
Q provides the developer with promise objects, which are future representations of values. Useful for time travelling.
Async provides the developer with asynchronous versions of control structures and aggregate operations.
An example from one attempt at a linter implementation demonstrates a potential unity among libraries:
function lint(files, callback) {
// Function which returns a promise.
var getMerged = merger('.jslintrc'),
// Result objects to invoke callback with.
results = [];
async.each(files, function (file, callback) {
fs.exists(file, function (exists) {
// Future representation of the file's contents.
var contentsPromise,
// Future representation of JSLINT options from .jslintrc files.
optionPromise;
if (!exists) {
callback();
return;
}
contentsPromise = q.nfcall(fs.readFile, file, 'utf8');
optionPromise = getMerged(path.dirname(file));
// Parallelize IO operations.
q.all([contentsPromise, optionPromise])
.spread(function (contents, option) {
var success = JSLINT(contents, option),
errors,
fileResults;
if (!success) {
errors = JSLINT.data().errors;
fileResults = errors.reduce(function (soFar, error) {
if (error === null) {
return soFar;
}
return soFar.concat({
file: file,
error: error
});
}, []);
results = results.concat(fileResults);
}
process.nextTick(callback);
})
.catch(function (error) {
process.nextTick(function () {
callback(error);
});
})
.done();
});
}, function (error) {
results = results.sort(function (a, b) {
return a.file.charCodeAt(0) - b.file.charCodeAt(0);
});
callback(error, results);
});
}
I want to do something potentially-blocking for each file. So async.each is the obvious choice. I can parallelize related operations per-iteration with q.all and reuse my option values if they apply to 2 or more files.
Here, Async and Q each influence the control flow of the program, and Q represents values resolving to file contents sometime in the future. The libraries work well together. One does not need to "choose one over the other".

Callback pyramids in your code can be simplified using promise composition and javascript lexical scoping.
removeItem: function (itemId) {
var removeRegexp = new RegExp('\\/stock\\.php\\?remove=' + itemId);
var found = false
var promise = getPage('/sock.php')
_.times(5, (i) => {
promise = promise.then((webpage) => {
if (found) return true
var removeMatch = webpage.match(removeRegexp)
var found = removeMath !== null
var nextPage = found ? removeMatch[0] : '/stock.php?p='+i+1
return Q.delay(1000).then(() => this.getPage(nextPage))
})
})
return promise.fail(console.log.bind(console))
},
IMHO async should not be used in new javascript code. Promises are more composable, and allow for a lot more intutive code.
The primary reason why node did not use promises was because of performance concerns which have largely been addressed very well by libraries like Bluebird and Q.
As async/await syntax becomes more mainstream, promises will pave the way for code that looks very similar with synchronous code.

While this is still not an actual answer to my question (Q vs async), regarding my problem, I've found Selenium / WebDriverJs to be a viable solution.
driver.get('http://www.google.com');
driver.findElement(webdriver.By.name('q')).sendKeys('webdriver');
driver.findElement(webdriver.By.name('btnG')).click();
driver.wait(function() {
return driver.getTitle().then(function(title) {
return title === 'webdriver - Google Search';
});
}, 1000);
WebDriver uses a queue to execute promises sequentially, which helps immensely with controlling indentation. Its promises are also compatible with Q's.
Creating a sequence of promises is no longer an issue. A simple for-loop will do.
As for stopping early in a sequence, don't do this. Instead of using a sequence, use an asynchronous-while design and branch.