We Keep Coding

Using findOne in a loop takes too long in Node.js

I'm using Node.js with MongoDB, I'm also using Monk for db access. I have the below code :
console.time("start");
collection.findOne({name: "jason"},
function(err, document) {
for(var i = 0; i < document.friends.length; i++) // "friends is an array contains ids of the user's friends"
{
collection.findOne({id: document.friends[i]}, function(err, doc)
{
console.log(doc.name);
});
}
});
console.log("The file was saved!");
console.timeEnd("start");
I have two questions regarding this code :
I see the execution time and "The file was saved!" string first, then I see the names of the friends coming in the console. Why is that? Shouldn't I see the names first then the execution time? Is it because the async nature of Node.js?
Names are printing very slowly in the console, the speed is like one name in two seconds. Why is it so slow? Is there a way to make the process faster?
EDIT:
Is it a good idea to break friends list to smaller pieces and call friends asynchronously? Would it make the process faster?
EDIT 2:
I changed my code to this :
collection.find({ id: { "$in": document.friends}}).then(function(err, doc)
{
console.log(doc.name);
if(err) {
return console.log(err);
}
}
This doesn't give an error, but this doesn't print anything either.
Thanks in advance.

Answer for question 1:
Yes, you are right.
Is it because the async nature of Node.js.
And to prevent that Node.js provides some mechanism for that you can use it otherwise you can do it on your own manually by setting one flag.
Answer for question 2:
you can use $in instead of findOne, it will be ease and fast.
e.g. .find({ "fieldx": { "$in": arr } })
arr :- In this you need to provide whole array.

yes, it's because javascript's async nature.
As you have called db from for loop javascript will not wait for it's response and continue the execution so it will print the file was saved first.
about your ans 2
It's making a dbCall for every friend then it's obvious that it will take some time that's why it's taking 1 or 2 secs for every friend.
console.time("start");
collection.findOne({name: "jason"},
function(err, document) {
for(var i = 0; i < document.friends.length; i++) // "friends is an array contains ids of the user's friends"
{
console.log("InsideforLoop Calling " + i + " friend");
collection.findOne({id: document.friends[i]}, function(err, doc)
{
console.log(doc.name);
});
console.log("Terminating " + i + "-----");
}
});
console.log("The file was saved!");
console.timeEnd("start");
This will make your async and db doubts more clear.
As you will see it will print all console in line.
InsideforLoop Calling 0 friend
Terminating 0 -----
and so on....Like this
console.log(doc.name);
but this will be printed asynchronusly
Added
collection.findOne({name: "jason"},
function(err, document) {
//you can do this
collection.find({id: $in:{document.friends}, function(err, doc)
{
console.log(doc);
});
});
Find All Details in one call

collection.aggregate([
{
$match:{
id :{ "$in" : document.friends},
}
}
]).exec(function ( e, d ) {
console.log( d )
if(!e){
// your code when got data successfully
}else{
// your code when you got the error
}
});

collection.findOne({name: "jason"},
function(err, document) {
if(document != undefined){
collection.find({ id: { "$in": document.friends}}).then(function(err, doc)
{
console.log(doc.name);
if(err) {
return console.log(err);
}
}
}
});

Answer to 1: Yes, it is because node is async. The part where it logs names is executed only when the first findOne returns, whereas the file was saved is executed straight away.

node.js - how to pass additional parameters to async.map

Background Information
I'm trying to write some javascript / node.js code that will do the following:
Query a redis database and get back a bunch of keys from my hash. This is what the command looks like on the redis-cli:
127.0.0.1:6379> hkeys widgets:1231231234
1) "13:00:00_17:00:00_mtwrf"
2) "00:00:00_00:00:00"
3) "08:00:00_12:00:00_mtwrf"
or
127.0.0.1:6379> hkeys widgets:2222222222
1) "00:00:00_00:00:00"
Each widget will have at least one key... called the default key. Default keys look like this: "00:00:00_00:00:00"
For each HKEYS query, if there's more than one result returned, I need to test each key returned (except the default) against a set of match criteria. Whichever key matches, is what is used to do a subsequent HGET against redis. So for example, in the first result set shown above... If key 3 was a match I would run the following command:
127.0.0.1:6379> hget widgets:2222222222 08:00:00_12:00:00_mtwrf
"some value"
127.0.0.1:6379>
If neither key 1 or 3 matches, then I query for the default key.
Code
I've recently discovered the async module. I'm currently using it in my code to loop through the results from HKEYS.
Please see the code below:
async.map Code
router.get('/:widgetnum', function(req, res, next) {
//validate widgetnum format
var widgetnum = req.params.widgetnum;
if ( !valid_widget(widgetnum) ) {
var retval = {"res":false, "msg":"malformed widgetnum"};
res.send(JSON.stringify(retval));
} else {
var keys = {};
redis.hkeys("widgets:" + widgetnum, function(err, data){
if (err) {
res.send(JSON.stringify(false));
}
if (data) {
current = getCurrentUTC(); // needed by iterator for match criteria. Using a global variable for now.
async.map(data, hash_iterator, function (err, iterator_results) {
if (iterator_results) {
console.log("it are: " + iterator_results);
}
res.send(iterator_results);
}); //end async map
}
}); //end redis.hkeys
}
});
Question / Problem
This is working in the sense that for each key returned by HKEYS, I'm able to run the "hash_iterator" function.
However, once inside the iterator function, after evaluation / running my match criteria on each result, I don't have all the information I need to run the secondary HGET query.
The code:
async.map(data, hash_iterator, function (err, iterator_results)
passes just the values:
"13:00:00_17:00:00_mtwrf"
"00:00:00_00:00:00"
"08:00:00_12:00:00_mtwrf"
But I need the hash name (in this case "widgets:" ) and the widgetnum to make the HGET call.
I guess I can use global variables... but I wanted to make sure that my approach in general is correct here.
Any input would be appreciated.
Here's the hash_iterator logic:
var hash_iterator = function (redis_key, doneCallBack) {
var results = {};
console.log("processing..." + redis_key);
//skip if default rule....
if (redis_key.indexOf('00:00:00_00:00:00') == -1){
//need to write logic here to do HGET and save
//results somewhere... in case no other keys match
} else {
// run some logic to test match criteria.
if (matchfound) {
console.log("bingo. found match in " + redis_key);
redis.hget(hashname + widgetnum + redis_key, function (e, results){
if (results) {
return doneCallBack(null, results);
} else {
return doneCallBack(null, null);
}
});
}else {
console.log ("no match");
}
}
return results;
}

You can solve it in functional style with map. Just generate array of pairs and then pass it to async.map, for example
from [1,2,3] you can generate [['key', '1'], ['key', '2'], ['key', 3]]
So here the code, that generating pairs:
redis.hkeys("widgets:" + widgetnum, function(err, data){
// some code goes here
var pairs = data.map(function(ts) {
return [`widgets:${widgetnum}`, ts];
});
// async.map call goes here
}); //end redis.hkey
Inside async.map iterator callback you can access key with iterator_results[0]
But in your case the variable widgetnums is not global variable. It's just variable from upper scope and you can use it inside nested functions if you define it in propper scope. It's very useful trick that called lexical closure.

Resource object returned instead of Mongoose.count value

I'm having a little trouble trying to retrieve Mongoose.count values from a function in my back end API within my mean.js application. Everything is routed correctly as far as I can see, and seems to be working absolutely fine until I get to the front end angular code and try to retrieve the data from the API via a service and $resource.
Here's my back end API function to retrieve the count of listings in a particular category, where the category is passed in correctly as parameter.
exports.getListingCountForSpecificCategory = function(req, res, next, category) {
Listing.count({
category: category,
quantityAvailable: { $gt: 0 },
listingActive: true
}, function(err, count){
if(err) {
console.log(err);
} else {
console.log(category + ': ' + count);
res.jsonp(count);
}
});
};
This runs correctly and within the console.log()
console.log(category + ': ' + count);
The category count is returned correctly. So, everything working correctly!
Until I get to retrieving the count value with angular on the front end. Here's the function I've written:
$scope.listingsCount = function(category) {
$scope.catCount = Listings.listingsCategoryCount.query({
category: category.alias
});
$scope.catCount.$promise.then(function(count){
category.listingCount = count;
});
};
When this function is run and the category object is passed into it, instead of it retrieving the count value e.g. 14, it seems to retrieve a resource promise object instead, with the count value nowhere to be seen. I've looked over the code a few times and can't for the life of me figure out why.
Here's the service I'm using, just in case you need to see it.
listingsCategoryCount: $resource('listingscount/:category', {
category: '#_category'
}, {
query: {
method: 'GET',
isArray: false
}
}),
It's a mystery to me why the count value isn't being returned. I may be going about this incorrectly of course. Any help would be greatly appreciated.
Thanks

https://docs.angularjs.org/api/ngResource/service/$resource:
On success, the promise is resolved with the same resource instance or collection object, updated with data from server.
Change this: res.jsonp(count); to res.jsonp({ count: count }); and it should work: you'll get a Resource object with property count.

Best method to string together variety of DB calls in Node js

I basically need to make about 3 calls to get the data for a json object.. It basically JSON array of JSON object which have some attributes, one of which is an array of other values selected using a second query, then that one also has an array inside which is selected with another db call.
I tried using asyn.concatSeries so that I can dig down into the bottom call and put together all the information I collected for one root json object but that's creating a lot of unexpected behaviour..
Example of JSON
[
{
"item" : "firstDbCall"
"children" : [ {
"name" : "itemDiscoveredWithSecondDBCall"
"children" : [ itemsDiscoveredwith3rdDBCall]
},
]
}
]
This is really difficult using node.js. I really need to figure out how to do this properly since I have to do many of these for different purposes.
EDIT
This is the code i have. There's some strange behaviour with async.concatSeries. The results get called multiple times after each one of the functions finish for each array. So i had to put a check in place. I know it's very messy code but i've been just putting band-aids all over it for the past 2 hours to make it work..
console.log("GET USERS HAREDQARE INFO _--__--_-_-_-_-_____");
var query = "select driveGroupId from tasks, driveInformation where agentId = '"
+ req.params.agentId + "' and driveInformation.taskId = tasks.id order by driveInformation.taskId desc;";
connection.query(query, function(err, rows) {
if (rows === undefined) {
res.json([]);
return;
}
if(rows.length<1) { res.send("[]"); return;}
var driveGroupId = rows[0].driveGroupId;
var physicalQuery = "select * from drives where driveGroupId = " + driveGroupId + ";";
connection.query(physicalQuery, function(err, rows) {
console.log("ROWSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS");
console.log(rows);
async.concatSeries(rows, function(row, cb) {
console.log("-------------------------------SINGLE ROW-------------------------------------");
console.log(row);
if(row.hasLogicalDrives != 0) {
console.log("HAS LOGICAL DRIVES");
console.log(row.id);
var query = "select id, name from logicalDrives where driveId = " + row.id;
connection.query(query, function(error, drives) {
console.log("QUERY RETURNED");
console.log(drives);
parseDriveInfo(row.name, row.searchable, drives, cb);
});
}
else
var driveInfo = { "driveName" : row.name, "searchable" : row.searchable};
console.log("NO SUB ITEMS");
cb(null, driveInfo);
}, function(err, results) {
console.log("GEETTTTINGHERE");
console.log(results);
if(results.length == rows.length) {
console.log("RESULTS FOR THE DRIVE SEARCH");
console.log(results);
var response = {"id": req.params.agentId};
response.driveList = results;
console.log("RESPONSE");
console.log(response);
res.json(response);
}
});
});
});
};
parseDriveInfo = function(driveName, searchable, drives, cb) {
async.concatSeries(drives, function(drive,callback) {
console.log("SERIES 2");
console.log(drive);
console.log("END OF DRIVE INFO");
var query = "select name from supportedSearchTypes where logicalDriveId = " + drive.id;
connection.query(query, function(error, searchTypes) {
drive.searchTypes = searchTypes;
var driveInfo = { "driveName" :driveName,
"searchable" : searchable,
"logicalDrives" : drive
};
callback(null, driveInfo);
});
}, function (err, results) {
console.log("THIS IS ISISIS ISISISSISISISISISISISISISIS");
console.log(results);
if(results.length === drives.length) {
console.log("GOTHERE");
cb(null, results);
}
});
}

Getting good enough with async to use exactly the right combination of methods under the right circumstances takes a fair amount of experience. Most likely your case in particular can be handled with async.waterfall if its query1 then query2(dataFoundByQuery1) then query3(dataFoundByQuery2). But depending on the circumstances you need to mix and match async methods appropriately and sometimes have 2 levels - for example a "big picture" async.waterfall where some of the steps in the waterfall do async.parallel or async.series as needed. I've never used async.concat and given your needs I think you have chosen the wrong method. The workhorses are async.each, async.eachSeries, async.waterfall, and async.map, at least for the web app & DB query use cases I mostly encounter, so make sure you really have those understood before exploring the more specific convenience methods.

EDIT: This is a more in depth example based on use of the connection library you seem to be using. Please note, some of this is javascript psuedo code. Things like adding objects to the resultsArray are clearly not complete, the only thing I took time to make sure was correct is the "flow of logic" as it pertains to callbacks. Everything else is for you to implement. In order to support multiple calls to the same callback function and maintain state from call to call, the best way is to wrap the set of callbacks in a closure. This allows the callbacks to share some state with the main event loop. This allows you to pass arguments to the callbacks, without actually having to pass them as arguments, much like class variables in c++, or even globals in javascript, but we haven't poluted the global scope :)
function queryDataBase(query) {
//wrap the whole query in a function so the callbacks can share some
//variables with similar scope. This is called a closure
int rowCounter = 0;
var dataRowsFromStep2;
var resultsArray = {};
connection.query(query, dataBaseQueryStep2);
function dataBaseQueryStep2(err, rows) {
//do something with err and rows
dataRowsFromStep2 = rows;
var query = getQueryFromRow(dataRowsFromStep2[rowCounter++]);//Always zero the first time. Might need to double check rows isn't empty!
connection.query(query, dataBaseQueryStep3);
}
function dataBaseQueryStep3(err, rows) {
//do something with err and rows
if(rowCounter < dataRowsFromStep2.size) {
resultsArray.add(rows);//Probably needs to be more interesting, but you get the idea
//since this is within the same closure, rowCounter maintains it's state
var query = getQueryFromRow(dataRowsFromStep2[rowCounter++]);
//recursive call query using dataBaseQueryStep3 as it's callback repeatedly until
//we run out of rows to call it on.
connection.query(query, dataBaseQueryStep3)
} else {
//when the if statement fails we have no more rows to run queries on so return to main program flow
returnToMainProgramLogic(resultsArray);
}
}
}
function returnToMainProgramLogic(results) {
//continue running your program here
}
I personally like the above logic better than the syntax async produces... I believe the heart of your problem rests in your nested calls to async, and the fact that ASYN itself, runs the series of functions asynchronously, but in order(confusing I know). If you write your program like this, you won't have to worry about it!

I would strongly suggest using sequelize.js It provides a really powerful orm that allows you to chain queries together. It also allows you to directly load your data into js objects, write dynamic sql, and connect to many different databases. Picture ActiveRecord from the Ruby world for Node.

Calling asynchronous function in callback

I'm having some trouble understanding asynchronous functions. I've read the chapter in Mixu's Node Book but I still can't wrap my head around it.
Basically I want to request a ressource (using the node package cheerio), parse it for valid URLs and add every match to my redis set setname.
The problem is that in the end it's only adding the first match to the redis set.
function parse(url, setname)
{
request(url, function (error, response, body)
{
if (!error && response.statusCode == 200)
{
$ = cheerio.load(body)
// For every 'a' tag in the body
$('a').each(function()
{
// Add blog URL to redis if not already there.
var blog = $(this).attr('href')
console.log("test [all]: " + blog);
// filter valid URLs
var regex = /http:\/\/[^www]*.example.com\//
var result = blog.match(regex);
if(result != null)
{
console.log("test [filtered]: " + result[0]);
redis.sismember(setname, result[0], function(err, reply)
{
if(!reply)
{
redis.sadd(setname, result[0])
console.log("Added " + result[0])
}
redis.quit()
})
}
})
}
})
}
I'd be very grateful for pointers on how I'd have to restructure this so the redis.sadd method is working with the correct result.
The output of the current implementation looks like:
test [all]: http://test1.example.com/
test [filtered]: http://test1.example.com/
...
Added http://test2.example.com/
So it's adding the test1.example.com but not printing the "added" line, and it's not adding the test2.example.com but it's printing the "added" line for it.
Thank you!

The first issue is caused by redis.sismember() being asynchronous: when its callback is called, you have already overwritten the result variable so it will point to the last value it had, and not the value at the moment at which you called redis.sismember().
One way to solve that is to create a new scoped variable by wrapping the asynchronous function in a closure:
(function(result) {
redis.sismember(setname, result[0], function(err, reply) {
...
});
})(result);
Another option is to create a partial function that's used as callback:
redis.sismember(setname, result[0], function(result, err, reply) {
...
}.bind(this, result));
The second issue is, I think, caused by redis.quit() being called, which closes the Redis connection after the first sadd(). You're not checking err, but if you do it might tell you more.