Background Information
I'm trying to write some javascript / node.js code that will do the following:
Query a redis database and get back a bunch of keys from my hash. This is what the command looks like on the redis-cli:
127.0.0.1:6379> hkeys widgets:1231231234
1) "13:00:00_17:00:00_mtwrf"
2) "00:00:00_00:00:00"
3) "08:00:00_12:00:00_mtwrf"
or
127.0.0.1:6379> hkeys widgets:2222222222
1) "00:00:00_00:00:00"
Each widget will have at least one key... called the default key. Default keys look like this: "00:00:00_00:00:00"
For each HKEYS query, if there's more than one result returned, I need to test each key returned (except the default) against a set of match criteria. Whichever key matches, is what is used to do a subsequent HGET against redis. So for example, in the first result set shown above... If key 3 was a match I would run the following command:
127.0.0.1:6379> hget widgets:2222222222 08:00:00_12:00:00_mtwrf
"some value"
127.0.0.1:6379>
If neither key 1 or 3 matches, then I query for the default key.
Code
I've recently discovered the async module. I'm currently using it in my code to loop through the results from HKEYS.
Please see the code below:
async.map Code
router.get('/:widgetnum', function(req, res, next) {
//validate widgetnum format
var widgetnum = req.params.widgetnum;
if ( !valid_widget(widgetnum) ) {
var retval = {"res":false, "msg":"malformed widgetnum"};
res.send(JSON.stringify(retval));
} else {
var keys = {};
redis.hkeys("widgets:" + widgetnum, function(err, data){
if (err) {
res.send(JSON.stringify(false));
}
if (data) {
current = getCurrentUTC(); // needed by iterator for match criteria. Using a global variable for now.
async.map(data, hash_iterator, function (err, iterator_results) {
if (iterator_results) {
console.log("it are: " + iterator_results);
}
res.send(iterator_results);
}); //end async map
}
}); //end redis.hkeys
}
});
Question / Problem
This is working in the sense that for each key returned by HKEYS, I'm able to run the "hash_iterator" function.
However, once inside the iterator function, after evaluation / running my match criteria on each result, I don't have all the information I need to run the secondary HGET query.
The code:
async.map(data, hash_iterator, function (err, iterator_results)
passes just the values:
"13:00:00_17:00:00_mtwrf"
"00:00:00_00:00:00"
"08:00:00_12:00:00_mtwrf"
But I need the hash name (in this case "widgets:" ) and the widgetnum to make the HGET call.
I guess I can use global variables... but I wanted to make sure that my approach in general is correct here.
Any input would be appreciated.
Here's the hash_iterator logic:
var hash_iterator = function (redis_key, doneCallBack) {
var results = {};
console.log("processing..." + redis_key);
//skip if default rule....
if (redis_key.indexOf('00:00:00_00:00:00') == -1){
//need to write logic here to do HGET and save
//results somewhere... in case no other keys match
} else {
// run some logic to test match criteria.
if (matchfound) {
console.log("bingo. found match in " + redis_key);
redis.hget(hashname + widgetnum + redis_key, function (e, results){
if (results) {
return doneCallBack(null, results);
} else {
return doneCallBack(null, null);
}
});
}else {
console.log ("no match");
}
}
return results;
}
You can solve it in functional style with map. Just generate array of pairs and then pass it to async.map, for example
from [1,2,3] you can generate [['key', '1'], ['key', '2'], ['key', 3]]
So here the code, that generating pairs:
redis.hkeys("widgets:" + widgetnum, function(err, data){
// some code goes here
var pairs = data.map(function(ts) {
return [`widgets:${widgetnum}`, ts];
});
// async.map call goes here
}); //end redis.hkey
Inside async.map iterator callback you can access key with iterator_results[0]
But in your case the variable widgetnums is not global variable. It's just variable from upper scope and you can use it inside nested functions if you define it in propper scope. It's very useful trick that called lexical closure.
Related
I have a an array of objects and each object contains another array. Value of Each object in the arrays is an id that needs to be replaced by the Name in the corresponding document with _id==id in the collection. In this particular case the async nature of NodeJS has been a pain and keeping track of whether the data of all the objects in the outer and inner arrays has been updated by finding them from the collection.
The easiest but dirty way of achieving this is with BLOCKING mongoose.find() call that would not execute the preceding statements until the data in the current object has been updated.
How can I achieve this?
Write now I am trying to do it like this but it's not working.
function getIDForName(name, callBackMethod) {
var _id = null;
var breakTheLoop = false;
const thread = spawn(function() {
// Everything we do here will be run in parallel in another execution context.
// Remember that this function will be executed in the thread's context,
// so you cannot reference any value of the surrounding code.
CircleTimeSong.getCircleTimeSongModelObject().find({songName:name}, function(err, foundData) {
console.error("callback");
if (err) {
console.error("error in getIDForName CircleTimeSongController.js: ", err);
//callBackMethod(null);
breakTheLoop = true;
}
console.log("found data length: ", foundData.length);
if (foundData != null && foundData.length > 0) {
console.log("doku 3", foundData[0]._id);
_id = foundData[0]._id;
breakTheLoop = true;
}
//callBackMethod(_id);
});
});
while(true) {
//console.log("in loop");
if(breakTheLoop)
break;
}
thread.kill();
return _id;
}
I am use nodejs npm package sql
I currently have an array of product skus like so..
var skus = ['product1', 'product2', 'product3'];
My sql store in a file as follows...
SELECT *
FROM stock AS s
WHERE s.sku IN (#skus)
Then I also have my prepared statement code as follows..
var connection = new sql.Connection(config, function(err) {
var ps = new sql.PreparedStatement(connection);
//Add params
if(params != undefined){
for(var key in params){
ps.input(key, sql.VarChar(200));
}
}
ps.prepare(sqlstatement, function(err) {
ps.execute(params, function(err, data) {
callback(null, data);
ps.unprepare(function(err) {
});
});
});
});
}
skus is contained correctly within the params object as the statement works fine when I am using it for simple WHERE X = #YI am just struggling with how I need pass the array of skus to allow them to work in the prepared statement.
I am amend the string using split and join to comma seperate them etc etc but I can't get these methods to work.
I assumed that I would need the param string to look like the following 'product1','product2','product3'.
would be also useful if someone could shed some light on how to debug the completed prepared statement so I can see what is actually being queried to SQL (with params inplace)
Many thanks in advance!
It appears that the sql object (i.e. the mssql module) has no attribute to handle arrays of anything. Moreover, specifying a scalar type in the call to ps.input similarly does not work.
The next best thing is to build keys for your array of parameters into your sql statement itself:
var connection = new sql.Connection(config, function(err) {
var ps = new sql.PreparedStatement(connection);
// Construct an object of parameters, using arbitrary keys
var paramsObj = params.reduce((obj, val, idx) => {
obj[`id${idx}`] = val;
ps.input(`id${idx}`, sql.VarChar(200));
return obj;
}, {});
// Manually insert the params' arbitrary keys into the statement
var stmt = 'select * from table where id in (' + Object.keys(paramsObj).map((o) => {return '#'+o}).join(',') + ')';
ps.prepare(stmt, function(err) {
ps.execute(paramsObj, function(err, data) {
callback(null, data);
ps.unprepare(function(err) {
});
});
});
});
}
I am trying to add hashtags in the post's hashtag[] array as a object with a num:1 variable to the users hashtagseen[] array if it is not already in it else add 1 the num if the hashtag is already in the hashtagseen[] array. How do I fix my code? Here is the code, thanks in advanced.
edit: I think I am not finding post.hashtag with this.hashtag and that is why it will not go to else. Just a guess.
The user object
Accounts.createUser({
username: username,
password: password,
email: email,
profile: {
hashtagsl:[],
}
});
collections/post.js
var post = _.extend(_.pick(postAttributes, 'title', 'posttext','hashtags'), {
userId: user._id,
username: user.username,
submitted: new Date().getTime(),
commentsCount: 0,
upvoters: [], votes: 0,
});
calling it
Meteor.call('addposthashtags',this.hashtags,Meteor.user().profile.hashtagsl);
lib/usershash
Meteor.methods({
addposthashtags: function (hashtags,hashtagsl) {
//supposed to make hashtagseen a array with the names from the hashtagsl object in it
var hashtagseen = _.pluck(hashtagsl, 'name');
//supposed to run once per each hashtag in the posts array.
for (var a = 0; a < hashtags.length; a++) {
//supposed set hashtagnumber to the number indexOf spits out.
var hashnumber=hashtagseen.indexOf(hashtags[a]);
//supposed to check if the current hashtag[a] === a idem in the hashtagseen.
if(hashnumber===-1){
var newhashtag = this.hashtags[a];
//supposed to make the object with a name = to the current hashtags
Meteor.users.update({"_id": this.userId},{"$push":{"profile.hashtagsl": {name: newhashtag, num: 1}}})
} else {
var hashi = hashtagseen[hashnumber];
//supposed to ad one to the num variable within the current object in hashtagsl
Meteor.users.update({"_id": this.userId, "profile.hashtagsl.name":hashi},{"$inc":{"profile.hashtagsl.num":1}});
}
}
}
});
Your addposthashtags function is full of issues. You also haven't provided a "schema" for hashtag objects.
addposthashtags: function () {
for (a = 0; a < this.hashtag.length; a++) {
// Issue1: You're querying out the user for every iteration of the loop!?
for (i = 0; i < Meteor.user().profile.hashtagseen.length; i++) {
// Issue2: You're comparing two _objects_ with ===
// Issue3: Even if you use EJSON.equals - the `num` property wont match
// Issue4: You're querying out the user again?
if (this.hashtag[a] === Meteor.user().profile.hashtagseen[i]) {
// Issue5 no `var` statement for hashtagseeni?
// Issue6 You're querying out the user again??
hashtagseeni = Meteor.user().profile.hashtagseen[i];
//Issue7 undefined hashtagsli?
//Issue8 Calling multiple methods for the one action (eg in a loop) is a waste of resources.
Meteor.call('addseen', hashtagsli);
} else {
//Issue9 no `var` statement for newhashtag?
newhashtag = this.hashtag[a];
newhashtag.num = 1;
//Issue8b Calling multiple methods for the one action (eg in a loop) is a waste of resources.
Meteor.call('updateUser', newhashtag, function (err, result) {
if (err)
console.log(err);
});
}
}
}
}
Also, the method has similiar issues:
addseen: function (hashtagseeni) {
// Issue10: var `profile` is undefined
// Issue11: should use `this.userId`
// Issue12: hashtagseeni wouldn't match profile.hashtagseen due to "num" field.
Meteor.users.update({"_id": Meteor.userId, "profile.hashtagseen": profile.hashtagseeni}, {"$inc":{"profile.hashtagseen.$.num":1}});
}
New issues with your new set of code:
Meteor.methods({
addposthashtags: function (hashtags,hashtagsl) {
//Issue1 `hashtag` is undefined, guessing you mean `hashtags`
//Issue2 no `var` for a
for (a = 0; a < hashtag.length; a++) {
//Issue3 no `var` for i
//Issue4 Why are you looping through both?
// don't you just want to check if hashtag[a] is in hashtagsl?
for (i = 0; i < hashtagsl.length; i++) {
if (hashtags[a] === hashtagsl[i].name) {
var hashi = hashtagsl[i].name;
//supposed to ad one to the num variable within the current object in hashtagsl.
// Issue5: This query wont do what you think. Test until you've got it right.
Meteor.users.update({"_id": Meteor.userId, 'profile.hashtagsl':hashi}, {"$inc":{"num":1}});
} else {
// Issue6 `this.hashtag` isn't defined. guessing you mean `hashtags[a]`
var newhashtag = this.hashtag[a];
// Issue7 superfluous statement
var newhashtagnum = num = 1;
// Issue8 Obvious syntax errors
// Perhaps try Meteor.users.update({"_id": this.userId},{"$push":{"profile.hashtagsl": {name: newhashtag, num: 1}}})
Meteor.users.update({"_id": Meteor.userId, 'profile'},{"$addToSet":{"hashtagsl"[newhashtag]=newhashtagnum}})
};
};
};
};
});
I'd suggest you trying the following
1) Assuming that after newhashtag=hashtag[a] you get a JSON object in newhashtag variable, try replacing newhashtag:{num:1}; with newhashtag.num = 1 - this will add the num variable to the object and set the value.
1.a) For debugging purposes try adding some console.log(JSON.stringify(newhashtag)); after each of the two lines where you're setting and changing the newhashtag variable - this way you'll know exactly what you're trying to add to the mongoDB document.
2) The update to increment the views also doesn't seem to me that will work. Couple of things to note here - $set:{'profile.hashtagseen[i]':num++} - MongoDB won't be able to identify the 'i' in 'profile.hashtagseen[i]' and 'num++' is not how increments are done in Mongo.
I'd suggest you look into the $inc and to the positional update documentation of MongoDB.
Your final increment update statement will look something like
Meteor.users.update({"_id": Meteor.userId, "profile.hashtagseen": profile.hashtagseen[i]}, {"$inc":{"profile.hashtagseen.$.num":1}});
I see that executing addposthashtags is in the client, and you must to pay attention because this function will execute in minimongo and doesn't work all operations. First you try execute this operation under mongo if it's work you must to create one function inside the folder server.
Add text of the documentation of Minimongo
In this release, Minimongo has some limitations:
$pull in modifiers can only accept certain kinds of selectors.
findAndModify, aggregate functions, and map/reduce aren't supported.
All of these will be addressed in a future release. For full Minimongo
release notes, see packages/minimongo/NOTES in the repository.
Minimongo doesn't currently have indexes. It's rare for this to be an
issue, since it's unusual for a client to have enough data that an
index is worthwhile.
You try create one method on the server, with the same operation.
Server:
Meteor.methods({
updateUser: function (newhashtag) {
Meteor.users.update(this.userId,
{
$addToSet: {'profile.$.hashtagseen': newhashtag}
});
}
});
Client:
Meteor.call('updateUser',newhashtag,function(err,result){
if (err)
console.log(err);// there you can print the erro if there are
});
Minimongo doesn't support alls operation, for test you can to execute in the console for testing the method if supported. After that you can to execute the operation under mongo directly, that clears your doubts.
I have only recently started developing for node.js, so forgive me if this is a stupid question - I come from Javaland, where objects still live happily sequentially and synchronous. ;)
I have a key generator object that issues keys for database inserts using a variant of the high-low algorithm. Here's my code:
function KeyGenerator() {
var nextKey;
var upperBound;
this.generateKey = function(table, done) {
if (nextKey > upperBound) {
require("../sync/key-series-request").requestKeys(function(err,nextKey,upperBound) {
if (err) { return done(err); }
this.nextKey = nextKey;
this.upperBound = upperBound;
done(nextKey++);
});
} else {
done(nextKey++);
}
}
}
Obviously, when I ask it for a key, I must ensure that it never, ever issues the same key twice. In Java, if I wanted to enable concurrent access, I would make make this synchronized.
In node.js, is there any similar concept, or is it unnecessary? I intend to ask the generator for a bunch of keys for a bulk insert using async.parallel. My expectation is that since node is single-threaded, I need not worry about the same key ever being issued more than once, can someone please confirm this is correct?
Obtaining a new series involves an asynchronous database operation, so if I do 20 simultaneous key requests, but the series has only two keys left, won't I end up with 18 requests for a new series? What can I do to avoid that?
UPDATE
This is the code for requestKeys:
exports.requestKeys = function (done) {
var db = require("../storage/db");
db.query("select next_key, upper_bound from key_generation where type='issue'", function(err,results) {
if (err) { done(err); } else {
if (results.length === 0) {
// Somehow we lost the "issue" row - this should never have happened
done (new Error("Could not find 'issue' row in key generation table"));
} else {
var nextKey = results[0].next_key;
var upperBound = results[0].upper_bound;
db.query("update key_generation set next_key=?, upper_bound=? where type='issue'",
[ nextKey + KEY_SERIES_WIDTH, upperBound + KEY_SERIES_WIDTH],
function (err,results) {
if (err) { done(err); } else {
done(null, nextKey, upperBound);
}
});
}
}
});
}
UPDATE 2
I should probably mention that consuming a key requires db access even if a new series doesn't have to be requested, because the consumed key will have to be marked as used in the database. The code doesn't reflect this because I ran into trouble before I got around to implementing that part.
UPDATE 3
I think I got it using event emitting:
function KeyGenerator() {
var nextKey;
var upperBound;
var emitter = new events.EventEmitter();
var requesting = true;
// Initialize the generator with the stored values
db.query("select * from key_generation where type='use'", function(err, results)
if (err) { throw err; }
if (results.length === 0) {
throw new Error("Could not get key generation parameters: Row is missing");
}
nextKey = results[0].next_key;
upperBound = results[0].upper_bound;
console.log("Setting requesting = false, emitting event");
requesting = false;
emitter.emit("KeysAvailable");
});
this.generateKey = function(table, done) {
console.log("generateKey, state is:\n nextKey: " + nextKey + "\n upperBound:" + upperBound + "\n requesting:" + requesting + " ");
if (nextKey > upperBound) {
if (!requesting) {
requesting = true;
console.log("Requesting new series");
require("../sync/key-series-request").requestSeries(function(err,newNextKey,newUpperBound) {
if (err) { return done(err); }
console.log("New series available:\n nextKey: " + newNextKey + "\n upperBound: " + newUpperBound);
nextKey = newNextKey;
upperBound = newUpperBound;
requesting = false;
emitter.emit("KeysAvailable");
done(null,nextKey++);
});
} else {
console.log("Key request is already underway, deferring");
var that = this;
emitter.once("KeysAvailable", function() { console.log("Executing deferred call"); that.generateKey(table,done); });
}
} else {
done(null,nextKey++);
}
}
}
I've peppered it with logging outputs, and it does do what I want it to.
As another answer mentions, you will potentially end up with results different from what you want. Taking things in order:
function KeyGenerator() {
// at first I was thinking you wanted these as 'class' properties
// and thus would want to proceed them with this. rather than as vars
// but I think you want them as 'private' members variables of the
// class instance. That's dandy, you'll just want to do things differently
// down below
var nextKey;
var upperBound;
this.generateKey = function (table, done) {
if (nextKey > upperBound) {
// truncated the require path below for readability.
// more importantly, renamed parameters to function
require("key-series-request").requestKeys(function(err,nKey,uBound) {
if (err) { return done(err); }
// note that thanks to the miracle of closures, you have access to
// the nextKey and upperBound variables from the enclosing scope
// but I needed to rename the parameters or else they would shadow/
// obscure the variables with the same name.
nextKey = nKey;
upperBound = uBound;
done(nextKey++);
});
} else {
done(nextKey++);
}
}
}
Regarding the .requestKeys function, you will need to somehow introduce some kind of synchronization. This isn't actually terrible in one way because with only one thread of execution, you don't need to sweat the challenge of setting your semaphore in a single operation, but it is challenging to deal with the multiple callers because you will want other callers to effectively (but not really) block waiting for the first call to requestKeys() which is going to the DB to return.
I need to think about this part a bit more. I had a basic solution in mind which involved setting a simple semaphore and queuing the callbacks, but when I was typing it up I realized I was actually introducing a more subtle potential synchronization bug when processing the queued callbacks.
UPDATE:
I was just finishing up one approach as you were writing about your EventEmitter approach, which seems reasonable. See this gist which illustrates the approach. I took. Just run it and you'll see the behavior. It has some console logging to see which calls are getting deferred for a new key block or which can be handled immediately. The primary moving part of the solution is (note that the keyManager provides the stubbed out implementation of your require('key-series-request'):
function KeyGenerator(km) {
this.nextKey = undefined;
this.upperBound = undefined;
this.imWorkingOnIt = false;
this.queuedCallbacks = [];
this.keyManager = km;
this.generateKey = function(table, done) {
if (this.imWorkingOnIt){
this.queuedCallbacks.push(done);
console.log('KG deferred call. Pending CBs: '+this.queuedCallbacks.length);
return;
};
var self=this;
if ((typeof(this.nextKey) ==='undefined') || (this.nextKey > this.upperBound) ){
// set a semaphore & add the callback to the queued callback list
this.imWorkingOnIt = true;
this.queuedCallbacks.push(done);
this.keyManager.requestKeys(function(err,nKey,uBound) {
if (err) { return done(err); }
self.nextKey = nKey;
self.upperBound = uBound;
var theCallbackList = self.queuedCallbacks;
self.queuedCallbacks = [];
self.imWorkingOnIt = false;
theCallbackList.forEach(function(f){
// rather than making the final callback directly,
// call KeyGenerator.generateKey() with the original
// callback
setImmediate(function(){self.generateKey(table,f);});
});
});
} else {
console.log('KG immediate call',self.nextKey);
var z= self.nextKey++;
setImmediate(function(){done(z);});
}
}
};
If your Node.js code to calculate the next key didn't need to execute an async operation then you wouldn't run into synchronization issues because there is only one JavaScript thread executing code. Access to the nextKey/upperBound variables will be done in sequence by only one thread (i.e. request 1 will access first, then request 2, then request 3 et cetera.) In the Java-world you will always need synchronization because multiple threads will be executing even if you didn't make a DB call.
However, in your Node.js code since you are making an async call to get the nextKey you could get strange results. There is still only one JavaScript thread executing your code, but it would be possible for request 1 to make the call to the DB, then Node.js might accept request 2 (while request 1 is getting data from the DB) and this second request will also make a request to the DB to get keys. Let's say that request 2 gets data from the DB quicker than request 1 and update nextKey/upperBound variables with values 100/150. Once request 1 gets its data (say values 50/100) then it will update nextKey/upperBound. This scenario wouldn't result in duplicate keys, but you might see gaps in your keys (for example, not all keys 100 to 150 will be used because request 1 eventually reset the values to 50/100)
This makes me think that you will need a way to sync access, but I am not exactly sure what will be the best way to achieve this.
I basically need to make about 3 calls to get the data for a json object.. It basically JSON array of JSON object which have some attributes, one of which is an array of other values selected using a second query, then that one also has an array inside which is selected with another db call.
I tried using asyn.concatSeries so that I can dig down into the bottom call and put together all the information I collected for one root json object but that's creating a lot of unexpected behaviour..
Example of JSON
[
{
"item" : "firstDbCall"
"children" : [ {
"name" : "itemDiscoveredWithSecondDBCall"
"children" : [ itemsDiscoveredwith3rdDBCall]
},
]
}
]
This is really difficult using node.js. I really need to figure out how to do this properly since I have to do many of these for different purposes.
EDIT
This is the code i have. There's some strange behaviour with async.concatSeries. The results get called multiple times after each one of the functions finish for each array. So i had to put a check in place. I know it's very messy code but i've been just putting band-aids all over it for the past 2 hours to make it work..
console.log("GET USERS HAREDQARE INFO _--__--_-_-_-_-_____");
var query = "select driveGroupId from tasks, driveInformation where agentId = '"
+ req.params.agentId + "' and driveInformation.taskId = tasks.id order by driveInformation.taskId desc;";
connection.query(query, function(err, rows) {
if (rows === undefined) {
res.json([]);
return;
}
if(rows.length<1) { res.send("[]"); return;}
var driveGroupId = rows[0].driveGroupId;
var physicalQuery = "select * from drives where driveGroupId = " + driveGroupId + ";";
connection.query(physicalQuery, function(err, rows) {
console.log("ROWSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS");
console.log(rows);
async.concatSeries(rows, function(row, cb) {
console.log("-------------------------------SINGLE ROW-------------------------------------");
console.log(row);
if(row.hasLogicalDrives != 0) {
console.log("HAS LOGICAL DRIVES");
console.log(row.id);
var query = "select id, name from logicalDrives where driveId = " + row.id;
connection.query(query, function(error, drives) {
console.log("QUERY RETURNED");
console.log(drives);
parseDriveInfo(row.name, row.searchable, drives, cb);
});
}
else
var driveInfo = { "driveName" : row.name, "searchable" : row.searchable};
console.log("NO SUB ITEMS");
cb(null, driveInfo);
}, function(err, results) {
console.log("GEETTTTINGHERE");
console.log(results);
if(results.length == rows.length) {
console.log("RESULTS FOR THE DRIVE SEARCH");
console.log(results);
var response = {"id": req.params.agentId};
response.driveList = results;
console.log("RESPONSE");
console.log(response);
res.json(response);
}
});
});
});
};
parseDriveInfo = function(driveName, searchable, drives, cb) {
async.concatSeries(drives, function(drive,callback) {
console.log("SERIES 2");
console.log(drive);
console.log("END OF DRIVE INFO");
var query = "select name from supportedSearchTypes where logicalDriveId = " + drive.id;
connection.query(query, function(error, searchTypes) {
drive.searchTypes = searchTypes;
var driveInfo = { "driveName" :driveName,
"searchable" : searchable,
"logicalDrives" : drive
};
callback(null, driveInfo);
});
}, function (err, results) {
console.log("THIS IS ISISIS ISISISSISISISISISISISISISIS");
console.log(results);
if(results.length === drives.length) {
console.log("GOTHERE");
cb(null, results);
}
});
}
Getting good enough with async to use exactly the right combination of methods under the right circumstances takes a fair amount of experience. Most likely your case in particular can be handled with async.waterfall if its query1 then query2(dataFoundByQuery1) then query3(dataFoundByQuery2). But depending on the circumstances you need to mix and match async methods appropriately and sometimes have 2 levels - for example a "big picture" async.waterfall where some of the steps in the waterfall do async.parallel or async.series as needed. I've never used async.concat and given your needs I think you have chosen the wrong method. The workhorses are async.each, async.eachSeries, async.waterfall, and async.map, at least for the web app & DB query use cases I mostly encounter, so make sure you really have those understood before exploring the more specific convenience methods.
EDIT: This is a more in depth example based on use of the connection library you seem to be using. Please note, some of this is javascript psuedo code. Things like adding objects to the resultsArray are clearly not complete, the only thing I took time to make sure was correct is the "flow of logic" as it pertains to callbacks. Everything else is for you to implement. In order to support multiple calls to the same callback function and maintain state from call to call, the best way is to wrap the set of callbacks in a closure. This allows the callbacks to share some state with the main event loop. This allows you to pass arguments to the callbacks, without actually having to pass them as arguments, much like class variables in c++, or even globals in javascript, but we haven't poluted the global scope :)
function queryDataBase(query) {
//wrap the whole query in a function so the callbacks can share some
//variables with similar scope. This is called a closure
int rowCounter = 0;
var dataRowsFromStep2;
var resultsArray = {};
connection.query(query, dataBaseQueryStep2);
function dataBaseQueryStep2(err, rows) {
//do something with err and rows
dataRowsFromStep2 = rows;
var query = getQueryFromRow(dataRowsFromStep2[rowCounter++]);//Always zero the first time. Might need to double check rows isn't empty!
connection.query(query, dataBaseQueryStep3);
}
function dataBaseQueryStep3(err, rows) {
//do something with err and rows
if(rowCounter < dataRowsFromStep2.size) {
resultsArray.add(rows);//Probably needs to be more interesting, but you get the idea
//since this is within the same closure, rowCounter maintains it's state
var query = getQueryFromRow(dataRowsFromStep2[rowCounter++]);
//recursive call query using dataBaseQueryStep3 as it's callback repeatedly until
//we run out of rows to call it on.
connection.query(query, dataBaseQueryStep3)
} else {
//when the if statement fails we have no more rows to run queries on so return to main program flow
returnToMainProgramLogic(resultsArray);
}
}
}
function returnToMainProgramLogic(results) {
//continue running your program here
}
I personally like the above logic better than the syntax async produces... I believe the heart of your problem rests in your nested calls to async, and the fact that ASYN itself, runs the series of functions asynchronously, but in order(confusing I know). If you write your program like this, you won't have to worry about it!
I would strongly suggest using sequelize.js It provides a really powerful orm that allows you to chain queries together. It also allows you to directly load your data into js objects, write dynamic sql, and connect to many different databases. Picture ActiveRecord from the Ruby world for Node.