We Keep Coding

Wait for a response to a Dexie table lookup before proceeding

I'm using a library called bugout (which is an API built on webtorrent) to create a P2P room but I need it to create the room based on a value from a table lookup using Dexie.
I know this has been rehashed a million times on Stack Overflow, but I'm still having trouble grasping the concept of promises or async await functions. I need the bugout operations to not proceed until this value is available. But I also don't want to get stuck in callback hell.
var room = db.profile.get(0, function (profile) {var ffname = profile.firstName;return ffname})
console.log(ffname) //returns undefined
var b = new Bugout(ffname);
I've also tried:
var room = db.profile.get(0, function (profile) {var ffname = profile.firstName;return ffname})
console.log(room) //returns undefined
var b = new Bugout(room);
How can I get ffname back with as little code as possible and not get stuck in a bunch of anonymous or asynchronous functions that will lock me out of the APIs?

The easiest and simplest way is this:
async function getBugoutFromId(id) {
var profile = await db.profile.get(id);
var ffname = profile.firstName;
console.log(ffname)
var b = new Bugout(ffname);
return b;
}
If you would prefer consuming promises without using async/await, do the following:
function getBugoutFromId(id) {
return db.profile.get(id).then(profile => {
var ffname = profile.firstName;
console.log(ffname)
var b = new Bugout(ffname);
return b;
});
}
The two functions works identically. Any of them returns a promise in its turn, so when you have called it. So whatever you are going to do with the retrieved Bugout instance, you need to handle the promise that your own function returns in the same manner as you did with the promise from Dexie.
async function someOtherFunction() {
var bugout = await getBugoutFromId(0);
console.log ("Yeah! I have a bugout", bugout);
}
or if you prefer not using async/await:
function someOtherFunction() {
return getBugoutFromId(0).then(bugout => {
console.log ("Year! I have a bugout", bugout);
});
}

Promises for promises that are yet to be created without using the deferred [anti]pattern

Problem 1: only one API request is allowed at a given time, so the real network requests are queued while there's one that has not been completed yet. An app can call the API level anytime and expecting a promise in return. When the API call is queued, the promise for the network request would be created at some point in the future - what to return to the app? That's how it can be solved with a deferred "proxy" promise:
var queue = [];
function callAPI (params) {
if (API_available) {
API_available = false;
return doRealNetRequest(params).then(function(data){
API_available = true;
continueRequests();
return data;
});
} else {
var deferred = Promise.defer();
function makeRequest() {
API_available = false;
doRealNetRequest(params).then(function(data) {
deferred.resolve(data);
API_available = true;
continueRequests();
}, deferred.reject);
}
queue.push(makeRequest);
return deferred.promise;
}
}
function continueRequests() {
if (queue.length) {
var makeRequest = queue.shift();
makeRequest();
}
}
Problem 2: some API calls are debounced so that the data to be sent is accumulated over time and then is sent in a batch when a timeout is reached. The app calling the API is expecting a promise in return.
var queue = null;
var timeout = 0;
function callAPI2(data) {
if (!queue) {
queue = {data: [], deferred: Promise.defer()};
}
queue.data.push(data);
clearTimeout(timeout);
timeout = setTimeout(processData, 10);
return queue.deferred.promise;
}
function processData() {
callAPI(queue.data).then(queue.deferred.resolve, queue.deferred.reject);
queue = null;
}
Since deferred is considered an anti-pattern, (see also When would someone need to create a deferred?), the question is - is it possible to achieve the same things without a deferred (or equivalent hacks like new Promise(function (resolve, reject) {outerVar = [resolve, reject]});), using the standard Promise API?

Promises for promises that are yet to be created
…are easy to build by chaining a then invocation with the callback that creates the promise to a promise represents the availability to create it in the future.
If you are making a promise for a promise, you should never use the deferred pattern. You should use deferreds or the Promise constructor if and only if there is something asynchronous that you want to wait for, and it does not already involve promises. In all other cases, you should compose multiple promises.
When you say
When the API call is queued, the promise for the network request would be created at some point in the future
then you should not create a deferred that you can later resolve with the promise once it is created (or worse, resolve it with the promises results once the promise settles), but rather you should get a promise for the point in the future at which the network reqest will be made. Basically you're going to write
return waitForEndOfQueue().then(makeNetworkRequest);
and of course we're going to need to mutate the queue respectively.
var queue_ready = Promise.resolve(true);
function callAPI(params) {
var result = queue_ready.then(function(API_available) {
return doRealNetRequest(params);
});
queue_ready = result.then(function() {
return true;
});
return result;
}
This has the additional benefit that you will need to explicitly deal with errors in the queue. Here, every call returns a rejected promise once one request failed (you'll probably want to change that) - in your original code, the queue just got stuck (and you probably didn't notice).
The second case is a bit more complicated, as it does involve a setTimeout call. This is an asynchronous primitive that we need to manually build a promise for - but only for the timeout, and nothing else. Again, we're going to get a promise for the timeout, and then simply chain our API call to that to get the promise that we want to return.
function TimeoutQueue(timeout) {
var data = [], timer = 0;
this.promise = new Promise(resolve => {
this.renew = () => {
clearTimeout(timer);
timer = setTimeout(resolve, timeout);
};
}).then(() => {
this.constructor(timeout); // re-initialise
return data;
});
this.add = (datum) => {
data.push(datum);
this.renew();
return this.promise;
};
}
var queue = new TimeoutQueue(10);
function callAPI2(data) {
return queue.add(data).then(callAPI);
}
You can see here a) how the debouncing logic is completely factored out of callAPI2 (which might not have been necessary but makes a nice point) and b) how the promise constructor only concerns itself with the timeout and nothing else. It doesn't even need to "leak" the resolve function like a deferred would, the only thing it makes available to the outside is that renew function which allows extending the timer.

When the API call is queued, the promise for the network request would
be created at some point in the future - what to return to the app?
Your first problem can be solved with promise chaining. You don't want to execute a given request until all prior requests have finished and you want to execute them serially in order. This is exactly the design pattern for promise chaining. You can solve that one like this:
var callAPI = (function() {
var p = Promise.resolve();
return function(params) {
// construct a promise that chains to prior callAPI promises
var returnP = p.then(function() {
return doRealNetRequest(params);
});
// make sure the promise we are chaining to does not abort chaining on errors
p = returnP.then(null, function(err) {
// handle rejection locally for purposes of continuing chaining
return;
});
// return the new promise
return returnP;
}
})();
In this solution, a new promise is actually created immediately with .then() so you can return that promise immediately - there is no need to create a promise in the future. The actual call to doRealNetRequest() is chained to this retrurned .then() promise by returning its value in the .then() handler. This works because, the callback we provide to .then() is not called until some time in the future when the prior promises in the chain have resolved, giving us an automatic trigger to execute the next one in the chain when the prior one finishes.
This implementation assumes that you want queued API calls to continue even after one returns an error. The extra few lines of code around the handle rejection comment are there to make sure the chain continues even where a prior promise rejects. Any rejection is returned back to the caller as expected.
Here's a solution to your second one (what you call debounce).
the question is - is it possible to achieve the same things without a
deferred (or equivalent hacks like new Promise(function (resolve,
reject) {outerVar = [resolve, reject]});), using the standard Promise
API?
As far as I know, the debouncer type of problem requires a little bit of a hack to expose the ability to trigger the resolve/reject callbacks somehow from outside the promise executor. It can be done a little cleaner than you propose by exposing a single function that is within the promise executor function rather than directly exposing the resolve and reject handlers.
This solution creates a closure to store private state that can be used to manage things from one call to callAPI2() to the next.
To allow code at an indeterminate time in the future to trigger the final resolution, this creates a local function within the promise executor function (which has access to the resolve and reject functions) and then shares that to the higher (but still private) scope so it can be called from outside the promise executor function, but not from outside of callAPI2.
var callAPI2 = (function() {
var p, timer, trigger, queue = [];
return function(data) {
if (!p) {
p = new Promise(function(resolve) {
// share completion function to a higher scope
trigger = function() {
resolve(queue);
// reinitialize for future calls
p = null;
queue = [];
}
}).then(callAPI);
}
// save data and reset timer
queue.push(data);
clearTimeout(timer);
setTimeout(trigger, 10);
return p;
}
})();

You can create a queue, which resolves promises in the order placed in queue
window.onload = function() {
(function(window) {
window.dfd = {};
that = window.dfd;
that.queue = queue;
function queue(message, speed, callback, done) {
if (!this.hasOwnProperty("_queue")) {
this._queue = [];
this.done = [];
this.res = [];
this.complete = false;
this.count = -1;
};
q = this._queue,
msgs = this.res;
var arr = Array.prototype.concat.apply([], arguments);
q.push(arr);
msgs.push(message);
var fn = function(m, s, cb, d) {
var j = this;
if (cb) {
j.callback = cb;
}
if (d) {
j.done.push([d, j._queue.length])
}
// alternatively `Promise.resolve(j)`, `j` : `dfd` object
// `Promise` constructor not necessary here,
// included to demonstrate asynchronous processing or
// returned results
return new Promise(function(resolve, reject) {
// do stuff
setTimeout(function() {
div.innerHTML += m + "<br>";
resolve(j)
}, s || 0)
})
// call `cb` here, interrupting queue
.then(cb ? j.callback.bind(j, j._queue.length) : j)
.then(function(el) {
console.log("queue.length:", q.length, "complete:", el.complete);
if (q.length > 1) {
q.splice(0, 1);
fn.apply(el, q[0]);
return el
} else {
el._queue = [];
console.log("queue.length:", el._queue.length
, "complete:", (el.complete = !el._queue.length));
always(promise(el), ["complete", msgs])
};
return el
});
return j
}
, promise = function(t) {
++t.count;
var len = t._queue.length,
pending = len + " pending";
return Promise.resolve(
len === 1
? fn.apply(t, t._queue[0]) && pending
: !(t.complete = len === 0) ? pending : t
)
}
, always = function(elem, args) {
if (args[0] === "start") {
console.log(elem, args[0]);
} else {
elem.then(function(_completeQueue) {
console.log(_completeQueue, args);
// call any `done` callbacks passed as parameter to `.queue()`
Promise.all(_completeQueue.done.map(function(d) {
return d[0].call(_completeQueue, d[1])
}))
.then(function() {
console.log(JSON.stringify(_completeQueue.res, null, 2))
})
})
}
};
always(promise(this), ["start", message, q.length]);
return window
};
}(window));
window
.dfd.queue("chain", 1000)
.dfd.queue("a", 1000)
.dfd.queue("b", 2000)
.dfd.queue("c", 2000, function callback(n) {
console.log("callback at queue index ", n, this);
return this
}, function done(n) {
console.log("all done callback attached at queue index " + n)
})
.dfd.queue("do", 2000)
.dfd.queue("other", 2000)
.dfd.queue("stuff", 2000);
for (var i = 0; i < 10; i++) {
window.dfd.queue(i, 1000)
};
window.dfd.queue.apply(window.dfd, ["test 1", 5000]);
window.dfd.queue(["test 2", 1000]);
var div = document.getElementsByTagName("div")[0];
var input = document.querySelector("input");
var button = document.querySelector("button");
button.onclick = function() {
window.dfd.queue(input.value, 0);
input.value = "";
}
}
<input type="text" />
<button>add message</button>
<br>
<div></div>

readable async codeflow in node js

I am new to async programming in node js and want to know how to translate my former (sync) programming to readable async programming in node js. Especially this example:
function getDefaultValue1() {
return getDefaultValue1FromDB();
};
function getDefaultValue2() {
return getDefaultValue2FromDB();
};
function doSomething(var optionalValue1, var optionalValue2){
try {
if(optionalValue1===null){
optionalValue1 = getDefaultValue1();
}
if(optionalValue2===null){
optionalValue2 = getDefaultValue2();
}
var resultList = [];
for(var i = 0; i < optionalValue2; i++) {
resultList.push(functionWithDbCall(optionalValue1, i));
}
return doSomeCalculations(resultList);
}catch(var e){
//log exception
return null
}
};
Note that all DB-Calls would be async in node js (getDefaultValue1FromDB(var callback), functionWithDbCall(var optionalValue1, var i, var callback)).
My thoughts are:
Adding callback-parameter to getDefaultValue => requires the same code in callback and after if. For loop exits before callbacks are called and callbacks need to handle how often they are called and to check to push the items in correct order. Changing a function to using an async method requires changing code everywhere it is called like changing getDefaultValue from returning a const int to a DB-call.
I am stuck here and all I found where advertising for some libraries or unreadable code worse then asm or regex (write once, read/touch never again). Did I miss something basic in my node js learning material or is this a design failure of the language?
So I hope somebody somebody can show me how to write this simple function readable with async node js.

I'd suggest using Bluebird:
var Promise = require('bluebird');
/**
* Lets assume getDefaultValue1FromDB() passes a callback fn;
* #return {Promise}
*/
function getDefaultValue1() {
return new Promise(function(resolve, reject){
return getDefaultValue1FromDB(function(err, value){
if (err) return reject(err);
return resolve(value);
});
});
}
/**
* Lets assume getDefaultValue2FromDB() passes a callback fn;
* #return {Promise}
*/
function getDefaultValue2() {
return new Promise(function(resolve, reject){
return getDefaultValue2FromDB(function(err, value){
if (err) return reject(err);
return resolve(value);
});
});
}
/**
* #param {Promise|int} optionalValue1
* #param {Prommise|int} optionalValue2
* #return {Promise}
*/
function doSomething(optionalValue1, optionalValue2){
if(optionalValue1 === null)
optionalValue1 = getDefaultValue1;
if(optionalValue2 === null)
optionalValue2 = getDefaultValue2;
return Promise
.all([optionalValue1, optionalValue2])
.spread(function(val1, val2){
var resultList = [];
for(var i = 0; i < val2; i++) {
resultList.push(functionWithDbCall(val1, i));
}
return doSomeCalculationsAsync(resultList);
}).catch(function(err){
//Log any errors here.
});
}
/**
* Calls the db and returns a promise with the result.
* #return {Promise} [description]
*/
function functionWithDbCall(val, i) {
Return new Promise();
}
/**
* #param {[Promise]} list Array of promises.
* #return {Promise}
*/
function doSomeCalculationsAsync(list) {
return Promise.all(list).then(doSomeCalculations);
}
Note that doSomething will return a promise, so you'll have to add a .then() to handle the results. Also asume that functionWithDbCall returns a Promise as well, you can then put all those together in a Promise.all() and then do the calculations.

You might want to use async/await from the ES7 proposal - a future version of JavaScript. See this answer on how to migrate from callbacks to promises to async/await.
To actually run future code today, you will need to compile it down to currently supported features with Babel. Plus it can do a lot more than async/await.

Angular $q Service - Limiting Concurrency for Array of Promises

Might help to give a bit of background context for this problem: I'm building an angular service that facilitates uploading chunks of multipart form data (mp4 video) to a storage service in the cloud.
I'm attempting to limit the number of unresolved promises (PUT requests of chunk data) happening concurrently. I am using $q.all(myArrayOfPromises).then()... to listen for all chunk upload promises being resolved, and then return an asynchronous call (POST to complete the file) when that happens. I think I'm encountering a race condition with my algorithm, because $q.all() gets called before all jobs have been scheduled for files with a lot of chunks, but succeeds for smaller files.
Here's my alogorithm.
var uploadInChunks = function (file) {
var chunkPromises = [];
var chunkSize = constants.CHUNK_SIZE_IN_BYTES;
var maxConcurrentChunks = 8;
var startIndex = 0, chunkIndex = 0;
var endIndex = chunkSize;
var totalChunks = Math.ceil(file.size / chunkSize);
var activePromises = 0;
var queueChunks = function () {
while (activePromises <= maxConcurrentChunks && chunkIndex < totalChunks) {
var deferred = $q.defer();
chunkCancelers.push(deferred); // array with broader scope I can use to cancel uploads as they're happening
var fileSlice = file.slice(startIndex, Math.min(endIndex, file.size));
chunkPromises.push(addChunkWithRetry(webUpload, chunkIndex, fileSlice).then(function () {
activePromises--;
queueChunks();
});
activePromises++;
startIndex += chunkSize;
endIndex += chunkSize;
chunkIndex++;
}
}
queueChunks();
return $q.all(chunkPromises).then(function () {
return filesApi.completeFile(file.fileId);
});
};
Even though $q.all is called prematurely, the chunks of the file that are still pending / not even scheduled at that time are eventually executed and resolved successfully.
I've done a fair amount of reading about throttling the concurrency of $q and know there are libraries out there to assist, but I'd really like to have an understanding of why this does not work all of the time :)

The promise that you're returning ($q.all) isn't really indicative of the promise you actually want to return. In your code above, the returned promise will finish after the first maxConcurrentChunks get resolved, because that's how many promises are in chunkPromises when you pass it to $q.all().
Another way to handle this (and get the result you want) would be the following psuedocode:
var uploadInChunks = function(file){
//...vars...
var fileCompleteDeferral = $q.defer();
var queueChunks = function(){
chunkPromises.push(nextChunk(chunkIndex).then(function () {
activePromises--;
if(allChunksDone()) { //could be activePromises == 0, or chunkIndex == totalChunks - 1
fileCompleteDeferral.resolve();
}
else {
queueChunks();
}
});
}
return fileCompleteDeferral.promise.then(completeFile());
}
The promise returned by this code will only resolve after ALL the promises are done, rather than just the first 8.

Node.js Asynchronous Library Comparison - Q vs Async

I have used kriskowal's Q library for a project (web scraper / human-activity simulator) and have become acquainted with promises, returning them and resolving/rejecting them, and the library's basic asynchronous control flow methods and error-throwing/catching mechanisms have proven essential.
I have encountered some issues though. My promise.then calls and my callbacks have the uncanny tendency to form pyramids. Sometimes it's for scoping reasons, other times it's to guarantee a certain order of events. (I suppose I might be able to fix some of these problems by refactoring, but going forward I want to avoid "callback hell" altogether.)
Also, debugging is very frustrating. I spend a lot of time console.log-ing my way to the source of errors and bugs; after I finally find them I will start throwing errors there and catching them somewhere else with promise.finally, but the process of locating the errors in the first place is arduous.
Also, in my project, order matters. I need to do pretty much everything sequentially. Oftentimes I find myself generating arrays of functions that return promises and then chaining them to each other using Array.prototype.reduce, which I don't think I should have to do.
Here is an example of one of my methods that uses this reduction technique:
removeItem: function (itemId) {
var removeRegexp = new RegExp('\\/stock\\.php\\?remove=' + itemId);
return this.getPage('/stock.php')
.then(function (webpage) {
var
pageCount = 5,
promiseFunctions = [],
promiseSequence;
// Create an array of promise-yielding functions that can run sequentially.
_.times(pageCount, function (i) {
var promiseFunction = function () {
var
promise,
path;
if (i === 0) {
promise = Q(webpage);
} else {
path = '/stock.php?p=' + i;
promise = this.getPage(path);
}
return promise.then(function (webpage) {
var
removeMatch = webpage.match(removeRegexp),
removePath;
if (removeMatch !== null) {
removePath = removeitemMatch[0];
return this.getPage(removePath)
.delay(1000)
// Stop calling subsequent promises.
.thenResolve(true);
}
// Don't stop calling subsequent promises.
return false;
}.bind(this));
}.bind(this);
promiseFunctions.push(promiseFunction);
}, this);
// Resolve the promises sequentially but stop early if the item is found.
promiseSequence = promiseFunctions.reduce(function (soFar, promiseFunction, index) {
return soFar.then(function (stop) {
if (stop) {
return true;
} else {
return Q.delay(1000).then(promiseFunction);
}
});
}, Q());
return promiseSequence;
}.bind(this))
.fail(function (onRejected) {
console.log(onRejected);
});
},
I have other methods that do basically the same thing but which are suffering from much worse indentation woes.
I'm considering refactoring my project using coalan's async library. It seems similar to Q, but I want to know exactly how they differ. The impression I am getting is that async more "callback-centric" while Q is "promise-centric".
Question: Given my problems and project requirements, what would I gain and/or lose by using async over Q? How do the libraries compare? (Particularly in terms of executing series of tasks sequentially and debugging/error-handling?)

Both libraries are good. I have discovered that they serve separate purposes and can be used in tandem.
Q provides the developer with promise objects, which are future representations of values. Useful for time travelling.
Async provides the developer with asynchronous versions of control structures and aggregate operations.
An example from one attempt at a linter implementation demonstrates a potential unity among libraries:
function lint(files, callback) {
// Function which returns a promise.
var getMerged = merger('.jslintrc'),
// Result objects to invoke callback with.
results = [];
async.each(files, function (file, callback) {
fs.exists(file, function (exists) {
// Future representation of the file's contents.
var contentsPromise,
// Future representation of JSLINT options from .jslintrc files.
optionPromise;
if (!exists) {
callback();
return;
}
contentsPromise = q.nfcall(fs.readFile, file, 'utf8');
optionPromise = getMerged(path.dirname(file));
// Parallelize IO operations.
q.all([contentsPromise, optionPromise])
.spread(function (contents, option) {
var success = JSLINT(contents, option),
errors,
fileResults;
if (!success) {
errors = JSLINT.data().errors;
fileResults = errors.reduce(function (soFar, error) {
if (error === null) {
return soFar;
}
return soFar.concat({
file: file,
error: error
});
}, []);
results = results.concat(fileResults);
}
process.nextTick(callback);
})
.catch(function (error) {
process.nextTick(function () {
callback(error);
});
})
.done();
});
}, function (error) {
results = results.sort(function (a, b) {
return a.file.charCodeAt(0) - b.file.charCodeAt(0);
});
callback(error, results);
});
}
I want to do something potentially-blocking for each file. So async.each is the obvious choice. I can parallelize related operations per-iteration with q.all and reuse my option values if they apply to 2 or more files.
Here, Async and Q each influence the control flow of the program, and Q represents values resolving to file contents sometime in the future. The libraries work well together. One does not need to "choose one over the other".

Callback pyramids in your code can be simplified using promise composition and javascript lexical scoping.
removeItem: function (itemId) {
var removeRegexp = new RegExp('\\/stock\\.php\\?remove=' + itemId);
var found = false
var promise = getPage('/sock.php')
_.times(5, (i) => {
promise = promise.then((webpage) => {
if (found) return true
var removeMatch = webpage.match(removeRegexp)
var found = removeMath !== null
var nextPage = found ? removeMatch[0] : '/stock.php?p='+i+1
return Q.delay(1000).then(() => this.getPage(nextPage))
})
})
return promise.fail(console.log.bind(console))
},
IMHO async should not be used in new javascript code. Promises are more composable, and allow for a lot more intutive code.
The primary reason why node did not use promises was because of performance concerns which have largely been addressed very well by libraries like Bluebird and Q.
As async/await syntax becomes more mainstream, promises will pave the way for code that looks very similar with synchronous code.

While this is still not an actual answer to my question (Q vs async), regarding my problem, I've found Selenium / WebDriverJs to be a viable solution.
driver.get('http://www.google.com');
driver.findElement(webdriver.By.name('q')).sendKeys('webdriver');
driver.findElement(webdriver.By.name('btnG')).click();
driver.wait(function() {
return driver.getTitle().then(function(title) {
return title === 'webdriver - Google Search';
});
}, 1000);
WebDriver uses a queue to execute promises sequentially, which helps immensely with controlling indentation. Its promises are also compatible with Q's.
Creating a sequence of promises is no longer an issue. A simple for-loop will do.
As for stopping early in a sequence, don't do this. Instead of using a sequence, use an asynchronous-while design and branch.