We Keep Coding

Using findOne in a loop takes too long in Node.js

I'm using Node.js with MongoDB, I'm also using Monk for db access. I have the below code :
console.time("start");
collection.findOne({name: "jason"},
function(err, document) {
for(var i = 0; i < document.friends.length; i++) // "friends is an array contains ids of the user's friends"
{
collection.findOne({id: document.friends[i]}, function(err, doc)
{
console.log(doc.name);
});
}
});
console.log("The file was saved!");
console.timeEnd("start");
I have two questions regarding this code :
I see the execution time and "The file was saved!" string first, then I see the names of the friends coming in the console. Why is that? Shouldn't I see the names first then the execution time? Is it because the async nature of Node.js?
Names are printing very slowly in the console, the speed is like one name in two seconds. Why is it so slow? Is there a way to make the process faster?
EDIT:
Is it a good idea to break friends list to smaller pieces and call friends asynchronously? Would it make the process faster?
EDIT 2:
I changed my code to this :
collection.find({ id: { "$in": document.friends}}).then(function(err, doc)
{
console.log(doc.name);
if(err) {
return console.log(err);
}
}
This doesn't give an error, but this doesn't print anything either.
Thanks in advance.

Answer for question 1:
Yes, you are right.
Is it because the async nature of Node.js.
And to prevent that Node.js provides some mechanism for that you can use it otherwise you can do it on your own manually by setting one flag.
Answer for question 2:
you can use $in instead of findOne, it will be ease and fast.
e.g. .find({ "fieldx": { "$in": arr } })
arr :- In this you need to provide whole array.

yes, it's because javascript's async nature.
As you have called db from for loop javascript will not wait for it's response and continue the execution so it will print the file was saved first.
about your ans 2
It's making a dbCall for every friend then it's obvious that it will take some time that's why it's taking 1 or 2 secs for every friend.
console.time("start");
collection.findOne({name: "jason"},
function(err, document) {
for(var i = 0; i < document.friends.length; i++) // "friends is an array contains ids of the user's friends"
{
console.log("InsideforLoop Calling " + i + " friend");
collection.findOne({id: document.friends[i]}, function(err, doc)
{
console.log(doc.name);
});
console.log("Terminating " + i + "-----");
}
});
console.log("The file was saved!");
console.timeEnd("start");
This will make your async and db doubts more clear.
As you will see it will print all console in line.
InsideforLoop Calling 0 friend
Terminating 0 -----
and so on....Like this
console.log(doc.name);
but this will be printed asynchronusly
Added
collection.findOne({name: "jason"},
function(err, document) {
//you can do this
collection.find({id: $in:{document.friends}, function(err, doc)
{
console.log(doc);
});
});
Find All Details in one call

collection.aggregate([
{
$match:{
id :{ "$in" : document.friends},
}
}
]).exec(function ( e, d ) {
console.log( d )
if(!e){
// your code when got data successfully
}else{
// your code when you got the error
}
});

collection.findOne({name: "jason"},
function(err, document) {
if(document != undefined){
collection.find({ id: { "$in": document.friends}}).then(function(err, doc)
{
console.log(doc.name);
if(err) {
return console.log(err);
}
}
}
});

Answer to 1: Yes, it is because node is async. The part where it logs names is executed only when the first findOne returns, whereas the file was saved is executed straight away.

Sqlite: check if database exist

I am developing a mobile application using phonegap that store some data into the local database (sqlite DB).
I need to know if the database exist or not, and that to determine which process need to execute.
var database = window.openDatabase("my_db", "1.0", "sample DB", 30000);
if (check_db_exist()) {
process_1();
}
else
{
process_2();
}

I needed to do something similar, I needed to check if the application had a db already created (legacy DB) and if so export all the data to the new db (new and improved DB) and then delete this DB.
Background Info: I was moving from simple keyvalue tables to complex relational DB with cascades etc.
function onDeviceReady() {
// CHECK IF LEGACY DATABASE EXISTS. IF DOES EXPORT EXISTING DATA TO NEW DB THEN DELETE LEGACY
window.resolveLocalFileSystemURL(cordova.file.applicationStorageDirectory + "/databases/<DatabaseName>.db", exportToNewDB, setupDB);
}
Note: If file exists (success), then we need to do our export code in here, then delete file so this method will always fail. If file doesn't exist - user has already exported to new DB or they our new user and never had legacy DB.
// Fail Method
function setupDB() {
newDB = window.sqlitePlugin.openDatabase({ name: "<NewImprovedDBName>.db" });
newDB.transaction(sql.initDB, sql.errorCallback, sql.successCallBack);
}
// Success Method
function exportToNewDB() {
db = window.sqlitePlugin.openDatabase({ name: "<LegacyDBName>.db" });
db.transaction(function (tx) {
setupDB();
// Insert Export code Here
// Delete DB
window.sqlitePlugin.deleteDatabase("<LegacyDBName>.db", sqlSuccess, sqlFail);
}, sqlFail);
}
Answer to your Question:
window.resolveLocalFileSystemURL(cordova.file.applicationStorageDirectory + "/databases/my_db.db", process_1, process_2);

The best way for determining if the DB exists or not is to check if the file that represents it exists. This is a simple IO operation, like the following example:
string path = Path.Combine(ApplicationData.Current.LocalFolder.Path, databaseName);
if (File.Exists(path))
{
//your code here
}

I don't think that you can check for the existence of the DB directly. I've researched and haven't found a way to do it using a simple function call. However, this seems to be a popular request, and here's a workaround:
var db = window.openDatabase("my.db", "1", "Demo", -1);
db.transaction(function (tx) {
/*I don't think that there is a way to check if a database exists.
As I think that the openDatabase call above will just create
a missing DB. Here is a second-best way to do it - there
is a way to check to see if a table exists. Just pick a table that
your DB should have and query it. If you get an ERROR, then you
can assume your DB is missing and you will need to create it. */
console.log("Trying to get the test data");
tx.executeSql("select * from test_table", [], function (tx, res) {
var len = res.rows.length, i;
for (i = 0; i < len; i++) {
console.log(res.rows.item(i).data);
}
}, function (err) {
console.log("Error selecting: " + err);
//NOW we need to create the DB and populate it
tx.executeSql('CREATE TABLE IF NOT EXISTS test_table (id integer primary key, data text)');
tx.executeSql('INSERT INTO test_table (data) VALUES("test data one")');
tx.executeSql('INSERT INTO test_table (data) VALUES("test data two")');
//now test select
tx.executeSql("select * from test_table", [], function (tx, res) {
var len = res.rows.length, i;
for (i = 0; i < len; i++) {
console.log(res.rows.item(i).data);
}
});
//now clean up so we can test this again
tx.executeSql('DROP TABLE IF EXISTS test_table', [], function (tx, res) {
console.log("dropped table");
}, function (err) {
console.log("Error dropping table: " + err);
});
});
});
As you can see, the error function during the first query will create and populate the DB. It's using exception handling for program logic flow, but it works!

W3C Web SQL Database transaction function

My question is based on W3C Web SQL Database. I am trying to populate SQL with the data I am getting from API. Here is my code
var db = window.openDatabase("Database", "1.0", "Cordova Demo", 200000);
for(i=0;i<data.responseMsg.length;i++){
var assignid_check = data.responseMsg[i].assignId;//------->3
db.transaction(queryDB, errorCB);//------->4
function errorCB(tx, err)
{
alert("Error processing SQL: "+err);
}
function queryDB(tx)
{
tx.executeSql('SELECT * FROM table_assign WHERE assign_id='+assignid_check, [], querySuccess, errorCB);
}
function querySuccess(tx, results) {
var len = results.rows.length;
if(len>0) //if exist jump to next array
{
alert("exist");
}
else //if not exist insert
{
tx.executeSql('INSERT INTO table_assign (audit_id,assign_id) VALUES (1,'+assignid_check+')');
alert("not exist");
}
}
}
Here problem is in line number 3 after getting "assignid_check" value I am calling transaction method. After transaction method I just want to call callback function i.e queryDB(tx) and if there is any error I want errorCB(tx,results) to get called. But soon after line 3 gets executed line 4 will execute and then again in for loop I value will get incremented and then it will execute line 3 and then 4. Only for the last value i.e only for last I value transaction will call queryDB(tx) callback function.
I am doing this for a phonegap application. Can someone help me on this?

how to pass arguments to PhoneGap database transaction executeSql functions

I'm new to PhoneGap I'm using database and its working fine with below code
db = window.openDatabase("Sample", "1.0", "PhoneGap Demo", 200000);
db.transaction(getDetails, transaction_error);
function transaction_error(tx, error) {
$('#busy').hide();
alert("Database Error: " + error);
}
function getUserDetails(tx) {
var sql = "select id, Name, DisplayName from details where Name=Name";
try {
tx.executeSql(sql, [],getList_success);
}
catch(err)
{
alert(err);
}
}
function getList_success(tx, results)
{
var len = results.rows.length;
for(var i=0; i <len; i++)
{
//Some code goes here
}
db = null;
}
Now i want to use the functions getUserDetails and getList_success by passing paramenters, i tried the below code its giving error
db = window.openDatabase("Sample", "1.0", "PhoneGap Demo", 200000);
getUserDetails(db);
function getUserDetails(tx) {
var sql = "select id, Name, DisplayName from details where Name=Name";
try {
tx.executeSql(sql, [],getList_success);
}
catch(err)
{
alert(err);
}
}
the error is TypeError:'undefined' is not a function in the catch block, can any one help me to get through this?.
Thanks in advance

The executeSql function need 4 parameters, try this:
function errorCallback(err){
//show error
}
...
tx.executeSql(sql, [],getList_success,errorCallback);

The problem is that you are handing over a callback function to db.transaction. If you want to provide some data to the callee you have to provide this in the context of the caller by using a closure.
var db = window.openDatabase("Sample", "1.0", "PhoneGap Demo", 200000);
var context = {val1: 'x', val2: 'y'};
db.transaction(function(tx){
myfunction(tx,context);
},
tx_error,
tx_success);
function myfunction(ty,context){
tx.executeSql(query,[context.val1,context.val2],myf_success,myf_error);
}
Keep in mind that this happens in a new thread! So if you call db.transaction be aware that this happens concurrently to the caller. If e.g. you are calling db.transaction() from inside the success callback of a jQuery.ajax() call the ajax thread will run on and the complete callback may be executed while the transaction is still in progress.

Look at your code:
db.transaction(getDetails, transaction_error);
getDetails!
Look at the functions you declared:
function getUserDetails(tx) {...}
function getList_success(tx, results) {....}
getDetails is not getUserDetails....
That's why you are getting the error.

NodeJs, javascript: .forEach seems to be asynchronous? need synchronization

I am currently working on a project with 3 friends using nodeJs, expressJs, MongoDB, html5,...
Since we're fairly new to these technologies we bumped into some problems.
A big problem that I can't find a solution for is the asynchronous execution of certain code.
I want a for each loop to finish, so that I have an updated online friends list, and than execute the res.render (in which I pass the online friends list), because currently it does the res.render before it finishes the loop.
Code:
function onlineFriends(req, res) {
var onlinefriends = new Array();
onlinefriends.push("mark");
FriendList.findOne({
owner: req.session.username
}, function (err, friendlist) {
friendlist.friends.forEach(function (friend) { // here forEach starts
OnlineUser.findOne({
userName: friend
}, function (err, onlineFriend) {
if (onlineFriend != null) {
onlinefriends.push(onlineFriend.userName);
console.log("a loop");
}
});
});
console.log("online friends: " + onlinefriends);
console.log("redirecting");
res.render('index', { // this is still inside the forEach function
friendlist: friendlist.friends,
onlinefriendlist: onlinefriends,
username: req.session.username
});// and here it ends
});
}
output will be as follows:
online friends: mark
redirecting
a loop
a loop
a loop
a loop
a loop
a loop
a loop
As discussed here ( JavaScript, Node.js: is Array.forEach asynchronous? ) , the answer is that the for-each is blocking, but in my example it seems to be non-blocking because it executes the res.render before it has finished looping?
How can I make sure that the for each is finished so I have an up to date onlinefriends list (and friendlist) which I can than pass to the res.render instead of the res.render happening way before the for -each loop finishes (which gives me an incorrect list of online users) ?
Thanks very much!

The following console log:
console.log("a loop");
is inside a callback
I believe that the callback of the function OnlineUser.findOne() is called asynchronously, that is why the code will log "a loop" after the redirect log
You should put the redirection after all the loop callbacks have been executed
Something like:
var count = 0;
friendlist.friends.forEach(function (friend) { // here forEach starts
OnlineUser.findOne({
userName: friend
}, function (err, onlineFriend) {
count++;
if (onlineFriend != null) {
onlinefriends.push(onlineFriend.userName);
console.log("a loop");
}
if(count == friendlist.friends.length) { // check if all callbacks have been called
redirect();
}
});
});
function redirect() {
console.log("online friends: " + onlinefriends);
console.log("redirecting");
res.render('index', { // this is still inside the forEach function
friendlist: friendlist.friends,
onlinefriendlist: onlinefriends,
username: req.session.username
});// and here it ends
}

I was able to solve something similar by adding the async package to my project and changing forEach() to async.each(). The advantage is that this provides a standard way to do synchronization for other parts of the application.
Something like this for your project:
function onlineFriends(req, res) {
var onlinefriends = new Array();
onlinefriends.push("mark");
FriendList.findOne({owner: req.session.username}, function (err, friendlist) {
async.each(friendlist.friends, function(friend, callback) {
OnlineUser.findOne({userName: friend}, function (err, onlineFriend) {
if (onlineFriend != null) {
onlinefriends.push(onlineFriend.userName);
console.log("a loop");
}
callback();
});
}, function(err) {
console.log("online friends: " + onlinefriends);
console.log("redirecting");
res.render('index', { // this is still inside the forEach function
friendlist: friendlist.friends,
onlinefriendlist: onlinefriends,
username: req.session.username
});
});
});
}

Running your code through jsbeautifier indents it properly and shows you why that happens:
function onlineFriends(req, res) {
var onlinefriends = new Array();
onlinefriends.push("mark");
FriendList.findOne({
owner: req.session.username
}, function (err, friendlist) {
friendlist.friends.forEach(function (friend) { // here forEach starts
console.log("vriend: " + friend);
OnlineUser.findOne({
userName: friend
}, function (err, onlineFriend) {
if (onlineFriend != null) {
onlinefriends.push(onlineFriend.userName);
console.log("online friends: " + onlinefriends);
}
});
console.log("nu door verwijzen");
res.render('index', { // this is still inside the forEach function
friendlist: friendlist.friends,
onlinefriendlist: onlinefriends,
username: req.session.username
});
}); // and here it ends
});
So... always indent your code properly and you won't have issues like this. Some editors such as Vim can indent your whole file with a single shortcut (gg=G in vim).
However, OnlineUser.findOne() is most likely asynchronous. so even if you move the call to the correct location it won't work. See ShadowCloud's answer on how to solve this.