We Keep Coding

Using promise in loop results in Promise failure

I'd like to reuse the same code in a loop. This code contains promises. However, when iterating, this code results in an error.
I've tried using for and while loops. There seems to be no issue when I use the for loop for a single iteration.
Here is a minimal version of my code:
var search_url = /* Some initial URL */
var glued = "";
for(var i = 0; i < 2; i++)
{
const prom = request(search_url)
.then(function success(response /* An array from a XMLHTTPRequest*/) {
if (/* Some condition */)
{
search_url = /* Gets next URL */
glued += processQuery(response[0]);
} else {
console.log("Done.")
}
})
.catch(function failure(err) {
console.error(err.message); // TODO: do something w error
})
}
document.getElementById('api-content').textContent = glued;
I expect the results to append to the variable glued but instead, I get an error: failure Promise.catch (async) (anonymous) after the first iteration of the loop.

Answer:
You can use the Symbol.iterator in accordance with for await to perform asynchronous execution of your promises. This can be packaged up into a constructor, in the example case it's called Serial (because we're going through promises one by one, in order)
function Serial(promises = []) {
return {
promises,
resolved: [],
addPromise: function(fn) {
promises.push(fn);
},
resolve: async function(cb = i => i, err = (e) => console.log("trace: Serial.resolve " + e)) {
try {
for await (let p of this[Symbol.iterator]()) {}
return this.resolved.map(cb);
} catch (e) {
err(e);
}
},
[Symbol.iterator]: async function*() {
this.resolved = [];
for (let promise of this.promises) {
let p = await promise().catch(e => console.log("trace: Serial[Symbol.iterator] ::" + e));
this.resolved.push(p);
yield p;
}
}
}
}
What is the above?
It's a constructor called Serial.
It takes as an argument an array of Functions that return Promises.
The functions are stored in Serial.promises
It has an empty array stored in Serial.resolved - this will store the resolved promise requests.
It has two methods:
addPromise: Takes a Function that returns a Promise and adds it to Serial.promises
resolve: Asynchronously calls a custom Symbol.iterator. This iterator goes through every single promise, waits for it to be completed, and adds it to Serial.resolved. Once this is completed, it returns a map function that acts on the populated Serial.resolved array. This allows you to simply call resolve and then provide a callback of what to do with the array of responses. A.e. .resolve()((resolved_requests) => //do something with resolved_requests)
Why does it work?
Although many people don't realize this Symbol.iterator is much more powerful than standard for loops. This is for two big reasons.
The first reason, and the one that is applicable in this situation, is because it allows for asynchronous calls that can affect the state of the applied object.
The second reason is that it can be used to provide two different types of data from the same object. A.e. You may have an array that you would like to read the contents of:
let arr = [1,2,3,4];
You can use a for loop or forEach to get the data:
arr.forEach(v => console.log(v));
// 1, 2, 3, 4
But if you adjust the iterator:
arr[Symbol.iterator] = function* () {
yield* this.map(v => v+1);
};
You get this:
arr.forEach(v => console.log(v));
// 1, 2, 3, 4
for(let v of arr) console.log(v);
// 2, 3, 4, 5
This is useful for many different reasons, including timestamping requests/mapping references, etc. If you'd like to know more please take a look at the ECMAScript Documentation: For in and For Of Statements
Use:
It can be used by calling the constructor with an Array of functions that return Promises. You can also add Function Promises to the Object by using
new Serial([])
.addPromise(() => fetch(url))
It doesn't run the Function Promises until you use the .resolve method.
This means that you can add promises ad hoc if you'd like before you do anything with the asynchronous calls. A.e. These two are the same:
With addPromise:
let promises = new Serial([() => fetch(url), () => fetch(url2), () => fetch(url3)]);
promises.addPromise(() => fetch(url4));
promises.resolve().then((responses) => responses)
Without addPromise:
let promises = new Serial([() => fetch(url), () => fetch(url2), () => fetch(url3), () => fetch(url4)])
.resolve().then((responses) => responses)
Data:
Since I can't really replicate your data calls, I opted for JSONPlaceholder (a fake online rest api) to show the promise requests in action.
The data looks like this:
let searchURLs = ["https://jsonplaceholder.typicode.com/todos/1",
"https://jsonplaceholder.typicode.com/todos/2",
"https://jsonplaceholder.typicode.com/todos/3"]
//since our constructor takes functions that return promises, I map over the URLS:
.map(url => () => fetch(url));
To get the responses we can call the above data using our constructor:
let promises = new Serial(searchURLS)
.resolve()
.then((resolved_array) => console.log(resolved_array));
Our resolved_array gives us an array of XHR Response Objects. You can see that here:
function Serial(promises = []) {
return {
promises,
resolved: [],
addPromise: function(fn) {
promises.push(fn);
},
resolve: async function(cb = i => i, err = (e) => console.log("trace: Serial.resolve " + e)) {
try {
for await (let p of this[Symbol.iterator]()) {}
return this.resolved.map(cb);
} catch (e) {
err(e);
}
},
[Symbol.iterator]: async function*() {
this.resolved = [];
for (let promise of this.promises) {
let p = await promise().catch(e => console.log("trace: Serial[Symbol.iterator] ::" + e));
this.resolved.push(p);
yield p;
}
}
}
}
let searchURLs = ["https://jsonplaceholder.typicode.com/todos/1", "https://jsonplaceholder.typicode.com/todos/2", "https://jsonplaceholder.typicode.com/todos/3"].map(url => () => fetch(url));
let promises = new Serial(searchURLs).resolve().then((resolved_array) => console.log(resolved_array));
Getting Results to Screen:
I opted to use a closure function to simply add text to an output HTMLElement.
This is added like this:
HTML:
<output></output>
JS:
let output = ((selector) => (text) => document.querySelector(selector).textContent += text)("output");
Putting it together:
If we use the output snippet along with our Serial object the final functional code looks like this:
let promises = new Serial(searchURLs).resolve()
.then((resolved) => resolved.map(response =>
response.json()
.then(obj => output(obj.title))));
What's happening above is this:
we input all our functions that return promises. new Serial(searchURLS)
we tell it to resolve all the requests .resolve()
after it resolves all the requests, we tell it to take the requests and map the array .then(resolved => resolved.map
the responses we turn to objects by using .json method. This is necessary for JSON, but may not be necessary for you
after this is done, we use .then(obj => to tell it to do something with each computed response
we output the title to the screen using output(obj.title)
Result:
let output = ((selector) => (text) => document.querySelector(selector).textContent += text)("output");
function Serial(promises = []) {
return {
promises,
resolved: [],
addPromise: function(fn) {
promises.push(fn);
},
resolve: async function(cb = i => i, err = (e) => console.log("trace: Serial.resolve " + e)) {
try {
for await (let p of this[Symbol.iterator]()) {}
return this.resolved.map(cb);
} catch (e) {
err(e);
}
},
[Symbol.iterator]: async function*() {
this.resolved = [];
for (let promise of this.promises) {
let p = await promise().catch(e => console.log("trace: Serial[Symbol.iterator] ::" + e));
this.resolved.push(p);
yield p;
}
}
}
}
let searchURLs = ["https://jsonplaceholder.typicode.com/todos/1", "https://jsonplaceholder.typicode.com/todos/2", "https://jsonplaceholder.typicode.com/todos/3"].map(url => () => fetch(url));
let promises = new Serial(searchURLs).resolve()
.then((resolved) => resolved.map(response =>
response.json()
.then(obj => output(obj.title))));
<output></output>
Why go this route?
It's reusable, functional, and if you import the Serial Constructor you can keep your code slim and comprehensible. If this is a cornerstone of your code, it'll be easy to maintain and use.
Using it with your code:
I will add how to specifically use this with your code to fully answer your question and so that you may understand further.
NOTE glued will be populated with the requested data, but it's unnecessary. I left it in because you may have wanted it stored for a reason outside the scope of your question and I don't want to make assumptions.
//setup urls:
var search_urls = ["https://jsonplaceholder.typicode.com/todos/1", "https://jsonplaceholder.typicode.com/todos/2"];
var request = (url) => () => fetch(url);
let my_requests = new Serial(search_urls.map(request));
//setup glued (you don't really need to, but if for some reason you want the info stored...
var glued = "";
//setup helper function to grab title(this is necessary for my specific data)
var addTitle = (req) => req.json().then(obj => (glued += obj.title, document.getElementById('api-content').textContent = glued));
// put it all together:
my_requests.resolve().then(requests => requests.map(addTitle));
Using it with your code - Working Example:
function Serial(promises = []) {
return {
promises,
resolved: [],
addPromise: function(fn) {
promises.push(fn);
},
resolve: async function(cb = i => i, err = (e) => console.log("trace: Serial.resolve " + e)) {
try {
for await (let p of this[Symbol.iterator]()) {}
return this.resolved.map(cb);
} catch (e) {
err(e);
}
},
[Symbol.iterator]: async function*() {
this.resolved = [];
for (let promise of this.promises) {
let p = await promise().catch(e => console.log("trace: Serial[Symbol.iterator] ::" + e));
this.resolved.push(p);
yield p;
}
}
}
}
//setup urls:
var search_urls = ["https://jsonplaceholder.typicode.com/todos/1", "https://jsonplaceholder.typicode.com/todos/2"];
var request = (url) => () => fetch(url);
let my_requests = new Serial(search_urls.map(request));
//setup glued (you don't really need to, but if for some reason you want the info stored...
var glued = "";
//setup helper function to grab title(this is necessary for my specific data)
var addTitle = (req) => req.json().then(obj => (glued += obj.title, document.getElementById('api-content').textContent = glued));
// put it all together:
my_requests.resolve().then(requests => requests.map(addTitle));
<div id="api-content"></div>
Final Note
It's likely that we will be seeing a prototypal change to the Promise object in the future that allows for easy serialization of Promises. Currently (7/15/19) there is a TC39 Proposal that does add a lot of functionality to the Promise object but it hasn't been fully vetted yet, and as with many ideas trapped within the Proposal stage, it's almost impossible to tell when they will be implemented into Browsers, or even if the idea will stagnate and fall off the radar.
Until then workarounds like this are necessary and useful( the reason why I even went through the motions of constructing this Serializer object was for a transpiler I wrote in Node, but it's been very helpful beyond that! ) but do keep an eye out for any changes because you never know!
Hope this helps! Happy Coding!

Your best bet is probably going to be building up that glued variable with recursion.
Here's an example using recursion with a callback function:
var glued = "";
requestRecursively(/* Some initial URL string */, function() {
document.getElementById('api-content').textContent = glued;
});
function requestRecursively(url, cb) {
request(url).then(function (response) {
if (/* Some condition */) {
glued += processQuery(response[0]);
var next = /* Gets next URL string */;
if (next) {
// There's another URL. Make another request.
requestRecursively(next, cb);
} else {
// We're done. Invoke the callback;
cb();
}
} else {
console.log("Done.");
}
}).catch(function (err) {
console.error(err.message);
});
}

Javascript callback function in blackboard pattern control flow

I have implemented a blackboard pattern in Javascript, my blackboard control iterates over knowledge sources / experts, and call their execAction().
for(let expert of this.blackboard.experts){
// Check execution condition
}
mostRelevantExpert.executeAction();
Now the problem is, those knowledge sources often need to call remote APIs or read files, and most of the libraries only provide callback APIs
class myExpert{
executeAction() {
myLibrary.call(params, (error, response) => { continueHere; })
}
}
Of course this is completely messing up the flow of my blackboard.
I am not sure whether the solution would be to reimplement the whole blackboard in an "asynchronous" fashion, or if there's a smarter way to go.
I've tried using libraries like deasync, but the problem is that I actually have a bug in myLibrary.call(params, (error, response) => { bueHere; } and I do not really understand now how to debug it. Since I am likely to have more problems like that in the future, was wondering what actions I should take.
Using node 6, ES6, and I don't like using callback programming style for what I'm doing here.
How should I go about the blackboard pattern in Javascript ?
How can I debug async code using node debug app.js
EDIT :
Here is my Blackboard Control code :
module.exports = class BlackboardControl{
constructor(blackboard){
this.blackboard = blackboard;
}
loop(){
console.log('¤ Blackboard Control');
console.log(' Starting Blackboard loop');
// Problem solved when there is a technicianAnswer, so the bot has something to say
while(!this.blackboard.problemSolved) {
// Select experts who can contribute to the problem
let candidates = [];
for(let expert of this.experts){
let eagerness = expert.canContribute();
if(eagerness){
candidates.push([eagerness,expert]);
}
}
if(candidates.length === 0) {
console.log('No expert can\'t do anything, returning');
return;
}
// Sort them by eagerness
candidates.sort(function(a,b) {
return a[0]-b[0];
});
for(let eagerExpert of candidates){
console.log('Next expert elected : ' + eagerExpert[1].constructor.name);
eagerExpert[1].execAction();
}
}
}
};

I haven't actually tried it out, yet (largely because I'd have to invent arbitrary problem spaces, and I feel like it would be much easier traveling in the other direction, right now)...
But if you want a look at what an async flow might look like, I might consider something like this:
async function getEngagedExperts (experts, problem) {
const getContributor = expert => expert.canContribute(problem)
.then(eagerness => [eagerness, expert]);
const contributors = await Promise.all(experts.map(getContributor));
return contributors.filter(([eager]) => eager);
}
async function contribute (previousState, expert) {
const state = await previousState;
return expert.execAction(state);
}
async function solveProblem (problem, experts) {
if (problem.solved) { return problem; }
const candidates = (await getEngagedExperts(experts, problem))
.sort(([a], [b]) => a - b)
.map(([, expert]) => expert);
const result = await candidates.reduce(contribute, Promise.resolve(problem));
return candidates.length ? solveProblem(result, experts) : undefined;
}
ES6 Generators + Yield
Works in ES6, if you have a library like co to manage the promises returned from iterators.
Writing your own co implementation is not that difficult, but this is totally not the space for it.
const getEngagedExperts = co.wrap(function * getEngagedExperts (experts, problem) {
const getContributor = expert => expert.canContribute(problem)
.then(eagerness => [eagerness, expert]);
const contributors = yield Promise.all(experts.map(getContributor));
return contributors.filter(([eager]) => eager);
});
const contribute = co.wrap(function * contribute (previousState, expert) {
const state = yield previousState;
return expert.execAction(state);
});
const solveProblem = co.wrap(function * solveProblem (problem, experts) {
if (problem.solved) { return problem; }
const candidates = (yield getEngagedExperts(experts, problem)))
.sort(([a], [b]) => a - b)
.map(([, expert]) => expert);
const result = yield candidates.reduce(contribute, Promise.resolve(problem));
return candidates.length ? solveProblem(result, experts) : undefined;
});
ES5 + Promise
When all else fails, write it by hand, in good ol' ES5, plus promises.
function getEngagedExperts (experts, problem) {
function getContributor (expert) {
return expert.canContribute(problem).then(eagerness => [eagerness, expert]);
}
function filterContributors (contributors) {
return contributors.filter(function (pair) {
const eagerness = pair[0];
return eagerness;
});
}
const getContributors = Promise.all(experts.map(getContributor));
return getContributors.then(filterContributors);
}
function contribute (previousComputation, expert) {
return previousComputation.then(function (state) {
return expert.execAction(state);
});
}
function solveProblem (problem, experts) {
if (problem.solved) { return problem; }
const getCandidates = getEngagedExperts(experts, problem)
.then(function (candidates) {
return candidates
.sort(function (a, b) { return a[0] - b[0]; })
.map(function (pair) { return pair[1]; });
});
return getCandidates.then(function (candidates) {
const getResult = candidates.reduce(contribute, Promise.resolve(problem));
return getResult.then(function (result) {
return candidates.length ? solveProblem(result, experts) : undefined;
});
});
}

Here is an attempt based on my (incomplete) understanding of your problem. These are the premises I used:
you have Expert objects that provide an asynchronous function that does some kind of work via the executeAction() method.
you have a BlackboardControl object that pools these experts and is responsible for running them in sequence until one of them returns a successful result. This object is also holding some kind of state encapsulated in the blackboard property.
The first step to a promise-based solution is to make the executeAction() method return a promise instead of requiring a callback. Changing the call convention of an entire node-style library is easily done with the promisifyAll() utility that Bluebird provides:
// module MyExpert ---------------------------------------------------------
var Promise = require('bluebird');
// dummy library with a node-style async function, let's promisify it
var myLibrary = Promise.promisifyAll({
someFunc: function (params, callback) {
setTimeout(() => {
if (Math.random() < 0.4) callback('someFunc failed');
else callback(null, {inputParams: params});
}, Math.random() * 1000 + 100);
}
});
class MyExpert {
executeAction(params) {
return myLibrary.someFuncAsync(params); // returns a promise!
}
}
module.exports = MyExpert;
now, we need a BlackboardControl object that does two things: pull out the next free Expert object from the pool (nextAvailableExpert()) and solve a given problem by applying experts to it in sequence, until one of them succeeds or a maximum retry count is reached (solve()).
// module BlackboardControl ------------------------------------------------
var Promise = require('bluebird');
var MyExpert = require('./MyExpert');
class BlackboardControl {
constructor(blackboard) {
this.blackboard = blackboard;
this.experts = [/* an array of experts */];
}
nextAvailableExpert() {
return new MyExpert();
// yours would look more like this
return this.experts
.map((x) => ({eagerness: x.canContribute(), expert: x}))
.filter((ex) => ex.eagerness > 0)
.sort((exA, exB) => exA.eagerness - exB.eagerness)
.map((ex) => ex.expert)
.pop();
}
solve(options) {
var self = this;
var expert = this.nextAvailableExpert();
if (!expert) {
return Promise.reject('no expert available');
} else {
console.info('Next expert elected : ' + expert.constructor.name);
}
options = options || {};
options.attempt = +options.attempt || 0;
options.maxAttempts = +options.maxAttempts || 10;
return expert.executeAction(/* call parameters here */).catch(error => {
options.attempt++;
console.error("failed to solve in attempt " + options.attempt + ": " + error);
if (options.attempt <= options.maxAttempts) return self.solve(options);
return Promise.reject("gave up after " + options.maxAttempts + " attempts.");
});
}
}
module.exports = BlackboardControl;
The key line is this one:
if (options.attempt <= options.maxAttempts) return self.solve(options);
Promises chain. If you return a new promise from a promise callback (in this case from the catch() handler, since we want to start over when an expert fails) the overall result of the promise will be determined by the result of this new promise. In other words, the new promise will be executed. This is our iterative step.
This way returning a promise from solve() enables internal repetition by simply calling solve() again in the error handler - and it enables reacting externally via then() as shown in below example usage:
// usage -------------------------------------------------------------------
var BlackboardControl = require('./BlackboardControl');
var bbControl = new BlackboardControl({ /* blackboard object */ });
var result = bbControl.solve({
maxAttempts: 10
}).then(response => {
console.log("continueHere: ", response);
}).catch(reason => {
console.error(reason);
});
which creates output like this (here dummy function happened to fail five times in a row):
Next expert elected : MyExpert
failed to solve in attempt 1: Error: someFunc failed
Next expert elected : MyExpert
failed to solve in attempt 2: Error: someFunc failed
Next expert elected : MyExpert
failed to solve in attempt 3: Error: someFunc failed
Next expert elected : MyExpert
failed to solve in attempt 4: Error: someFunc failed
Next expert elected : MyExpert
failed to solve in attempt 5: Error: someFunc failed
Next expert elected : MyExpert
continueHere: { some: 'parameters' }
During expert runs control is returned to the main program. Due to the fact that now multiple experts can run at the same time on multiple problems we can't make a list of available experts up-front. We must make a fresh decision every time we need an expert, hence the nextAvailableExpert() function.

Ah right, I actually managed to make the deasync code work. It turns out I was trying to use
const deasync = require('deasync');
try {
const deasyncAnswer = deasync(Lib.foo(
myParam,
// Callback was here
);
}
But the correct way to use it was
const fooDeasynced= deasync(Lib.foo);
try {
const deasyncAnswer = fooDeasynced(myparams)
}

know when async foreach inside async for each has finished

I'm using the async library in node.js, and I have an async.forEach call inside another async.forEach call and I need to know when the async calls have completely finished processing.
Code example:
var content = '...';
var resultArr = [];
async.each(regexArr, function(regex) {
var matches = content.match(regex);
async.each(matches, function(match) {
var result = regex.exec(match);
regex.lastIndex = 0;
resultArr.push(result[1]);
});
}, function(err) {
// I need to know when ALL inner async.forEach finished processing
// for ALL outside async.forEach and then do something with the resultArr
});

You can force synchronization by using async.map and an additional callback to handle the collection of inner callback results.
ALTHOUGH, with JS, execution is on a single thread, as other people have said, so I've included the MUCH simpler synchronous version first.
// Synchronous version is just as good for Regex stuff.
var content = '...';
var resultArr = [].concat.apply([], regexArr.map(function(regex) {
return (content.match(regex) || []).map(function(match) {
return (regex.exec(match) || [])[1];
});
}));
// IN THIS EXAMPLE, SYNCHRONOUS WOULD WORK TOO...
var content = '...';
collectRegexResults(content, regexArr, function(err, flatRegexResults) {
// Handle errors.
// flatRegexResults is equivalent to resultArr
// from your question.
});
function collectSingleRegexFor(content) {
return function(regex, passBackGroup) {
var matches = content.match(regex);
// Note use of || [] if match is null
async.map(matches || [], function(match, finishSingle) {
try {
finishSingle(null, regex.exec(match)[1]);
} catch (err) {
finishSingle(err, null);
}
}, passBackGroup); // Submits to outer as regexSets element
};
}
function collectRegexResults(text, regexArr, doStuffPostComplete) {
async.map(regexArr, collectSingleRegexFor(text), function(err, regexSets) {
if (err) {
return doStuffPostComplete(err);
}
// Flattening the Array again.
doStuffPostComplete(null, [].concat.apply([], regexSets));
});
}

sync independet callbacks result

I am searching for an elegant way to sync indepentent callbacks result invoked in unknown order.
function callback1() {
var result;
};
function callback2() {
var result;
};
//When done then call
function success(res1, res2) {
// do whatever
}
I know I can do something like:
var res = {};
var dfd = $.Deferred();
function callback1() {
var result;
res.res1 = result;
(res.res1 && res.res2) && (dfd.resolve(res));
};
function callback1() {
var result;
res.res2 = result;
(res.res1 && res.res2) && (dfd.resolve(res));
};
dfd.done(function(result){
// do whatever
});
but I would appreciate if somebody comes up with more elegant solution

If you return promises (builtin promises, not jQuery deferreds) and you don't care about order, then you can use Promise.all:
function callback1() {
return Promise.resolve(1)
}
function callback2() {
return Promise.resolve(2)
}
var ps = [callback1(), callback2()]
function add(x, y) {
return x + y
}
Promise.all(ps).then(function(result) {
return result.reduce(add)
}).then(console.log) // => 3
If you want to sequence them you can do it in such a way that you can apply a curried function that expects as many arguments as resolved promises there are by lifting it into the promise world. In other words:
function apply(pa, pf) {
return pf.then(function(f) {
return pa.then(f)
})
}
function lift(f, ps) {
return ps.reduce(function(pa, pb) {
return apply(pb, pa)
}, Promise.resolve(f))
}
function add(x) {
return function(y) {
return x + y
}
}
lift(add, ps).then(console.log) //=> 3
You can also sequence them in such a way that you don't need a curried function, by collecting the results in an array first then reducing it:
function sequence(ps) {
return ps.reduceRight(function(pa, pb) {
return pa.then(function(a) {
return pb.then(function(b) {
return [b].concat(a)
})
})
}, Promise.resolve([]))
}
function add(x, y) {
return x + y
}
// This looks similar to the Promise.all approach
// but they run in order
sequence(ps).then(function(result) {
return result.reduce(add)
}).then(console.log) // => 3

There are libraries that do this, such as the async library, but here's a "from scratch" solution. I'm also avoiding promises to avoid overwhelming you, but you should really read about them as they are the most elegant solution, albeit complicated for first timers.
function runInParallel(jobs, done) {
// Store all our results in an array.
var results = [];
// If one job fails, set this to true and use it to
// ignore all job results that follow.
var failureOccurred = false;
// Iterate over each of our registered jobs.
jobs.forEach(function (runJob, index) {
// Create a jobDone callback to pass to the job.
var jobDone = function (err, result) {
// If another job failed previously, abort processing
// this job's result. We no longer care.
if (failureOccurred) return;
// If this job passed in an error, set failure to true
// and pass the error to the final done callback.
if (err) {
failureOccurred = true;
done(err);
return;
}
// If we made it this far then push the job result into
// the results array at the same position as the job in
// the jobs array.
results[index] = result;
// If the results array contains as many results as the
// jobs array had jobs then we have finished processing
// them all. Invoke our done callback with an array of
// all results.
if (results.length === jobs.length) {
done(null, results);
}
};
// Begin the job and pass in our jobDone callback.
runJob(jobDone);
});
}
This will call all of your job functions in the array, passing in a jobDone callback the job should call when finished. If any job passes an error in then the function will immediately invoke the result callback with the error and ignore everything else. If the jobs succeed then you'll end up with an array of job results in the same positions as the jobs were in the jobs array. Simply modify your job functions to accept the jobDone callback.
var jobs = [
function job1(done) {
try {
var result;
done(null, result);
} catch (err) {
done(err);
}
},
function job2(done) {
try {
var result;
done(null, result);
} catch (err) {
done(err);
}
}
];
runInParallel(jobs, function (err, results) {
if (err) {
console.error(err);
return;
}
// results[0] = jobs[0] result
// results[1] = jobs[1] result
// etc...
});
Instead of an array of jobs you could modify this code to accept an object with property names. Then instead of assigning the results to the same position as the jobs in the jobs array you could assign the results to an object using the same property names.
Example (without comments this time):
function runInParallel(jobs, done) {
var results = {};
var failureOccurred = false;
Object.keys(jobs).forEach(function (jobName) {
var jobDone = function (err, result) {
if (failureOccurred) return;
if (err) {
failureOccurred = true;
done(err);
return;
}
results[jobName] = result;
if (results.length === jobs.length) {
done(null, results);
}
};
jobs[jobName](jobDone);
});
}
Then you can consume it like this:
var jobs = {
job1: function (done) {
try {
var result;
done(null, result);
} catch (err) {
done(err);
}
},
job2: function (done) {
try {
var result;
done(null, result);
} catch (err) {
done(err);
}
}
};
runInParallel(jobs, function (err, results) {
if (err) {
console.error(err);
return;
}
// results.job1 = job1 result
// results.job2 = job2 result
// etc...
});
The parallel function in the async library does almost exactly what we've done above. It even accepts an array of jobs or an object of named jobs like we did :)

Assuming your tasks (callback1() and callback2()) are synchronous, you might choose to write a reusable generalisation of the code in the question, in the form of a function that returns a function, trapping a couple of private vars in a closure :
function resultAggregator(n, fn) {
var results = {},
count = 0;
return function(id, res) {
count++;
results[id] = res;
if (count == n) {
fn(results);
}
}
}
So after calling resultAccumulator(), you have a function that can kept in-scope of other functions or passed to other parts of your code base. It makes no assumptions about ids or the nature of the results, except that they are synchronously derived. It will fire its callback when n results have been delivered.
var myResults = resultAggregator(2, function(results) {
// do whatever;
});
//The following commands may be in different parts of your code base
myResults('id1', synchTask1());
...
myResults('id2', synchTask2());
...
myResults('id3', synchTask3());
//The second tasks to deliver its data (ostensibly `synchTask1()` and `synchTask2()`, but not necessarily) will trigger the callback.
Demo
This is just one way to perform result aggregation. You might do something different depending on the exact scenario. Here's a slightly different formulation which records the order in which the results arrived :
Demo
Whatever you write, Deferreds/Promises are not necessary for the aggregation of synchronously derived data.
However, if any one task is, or may be, asynchronous then you may need a promise aggregator, eg jQuery.when() or Promise.all(), somewhere in the pattern.

Recursively traverse tree in Javascript

This is super simple task to do in Java but the asynchronous nature of javascript makes this task(for me) almost impossible, at least with my knowledge now.(I'm not trying to bash javascript. Love the language!).
It's very basic. A top level tree has a parent of null in my mysql database. It's easy finding children. The children have lines available to them. The depth of the tree is variable.
private static Set<Tree> getBranches( Tree trunk ) {
Set<Tree> treeSet = new HashSet<Tree>();
if ( trunk != null ) {
if ( trunk.hasLines() ) { //queries if tree has lines. returns true or false
treeSet.add( trunk );
}
for ( Tree tree : trunk.treeList ) {
treeSet.addAll( getBranches( tree ) );
}
}
return treeSet;
}
Basically the method tests if the tree has lines available. If it does it adds all of those to a set. If not it continues until it finds lines.
The asynchronous nature of the mysql node library turns this task into hell.
Here is what I have now
function hasLines(tree_id, callback) {
var ret;
pool.query('SELECT * from pkg_line_tree where tree_id = ?', [tree_id], function (err, rows) {
if (rows.length > 0) {
ret = true;
} else {
ret = false;
}
callback(ret);
});
}
function dig(tree_id, treeArray, callback) {
pool.query('SELECT * from tree where parent_id = ?', [tree_id], function (err, rows) {
if (rows) {
for (var i in rows) {
hasLines(rows[i].tree_id, function (t) {
if (t) {
treeArray.push(rows[i].tree_id);
} else {
treeArray.concat(dig(rows[i].tree_id, treeArray));
}
});
}
if (callback) {
callback(treeArray);
}
}
});
return treeArray;
}
var treeArray = [];
dig(52, treeArray, function (t) {
res.json(t);
});
I really just need to output all the children available in this root tree.
Please let me know if this doesn't make sense. I'll try to refactor. I'm hoping I got some kind of point across. I'd hate to use something like Fibers to get this done but I'm out of options. Thanks.

Your use of dig() isn't currently consistent:
// asynchronous with callback
dig(52, treeArray, function (t) {
res.json(t);
});
// then synchronous with `return`?
treeArray.concat(dig(rows[i].tree_id, treeArray));
Also, the concat in the last line isn't actually doing much, since it doesn't alter the array it's called on. You probably wouldn't actually want it to as dig passes around the treeArray rather than defining a new treeSet like in getBranches. So, if it did, it would append treeArray onto the end of itself each time.
You could still use concat with multiple treeSets, but you'd have to store its return value:
treeSet = treeSet.concat(subSet);
And, you'll have to replace the for loop this with an asynchronous iterator as the loop won't wait for asynchronous operations before continuing. The async library has a few options for this, if you're up for trying it.
So, with multiple treeSets, concat, and async.forEachSeries, you could try:
function dig(tree_id, callback) {
var treeSet = [];
hasLines(tree_id, function (yep) {
if (yep) {
treeSet.push(tree_id);
}
pool.query('SELECT * from tree where parent_id = ?', [tree_id], function (err, rows) {
function each(row, next) {
dig(row.tree_id, function (subSet) {
treeSet = treeSet.concat(subSet);
next(null);
});
}
function done() {
callback(treeSet);
}
async.forEachSeries(rows, each, done);
});
});
}
dig(52, function (treeSet) {
res.json(treeSet);
});

you have to use async https://github.com/caolan/async
I have modified your dig function to use async's forEach method
function dig(tree_id, treeArray, AllDone) {
pool.query('SELECT * from tree where parent_id = ?', [tree_id], function (err, rows) {
if (rows) {
async.forEach(
rows,
function(row, callback) {
hasLine(row.tree_id, function(t){
if (t) {
treeArray.push(row.tree_id);
callback();
}
else {
dig(row.tree_id, treeArray, callback);
}
});
},
function(err) {
if (err) AllDone(err, treeArray);
else AllDone(null, treeArray);
});
}
else
AllDone(null, treeArray)
});
}
treeArray = [];
dig(52, treeArray, function(err, t) {
res.json(t);
});
assuming rows is an array.. forEach go through each row and perform hasLine, each iteration will call the callback function when it finish, and AllDone will be called when all callback functions are called. the tricky part here is the recursion, each recursive call will have a forEach loop, and it will call the AllDone method only when all callbacks are finish.
however forEach execute in parallel, so order is not perserved
I think this should work, if you don't care about order.
Edit : you can use forEachSeries to solve the order problem.