I am currently working on a project with 3 friends using nodeJs, expressJs, MongoDB, html5,...
Since we're fairly new to these technologies we bumped into some problems.
A big problem that I can't find a solution for is the asynchronous execution of certain code.
I want a for each loop to finish, so that I have an updated online friends list, and than execute the res.render (in which I pass the online friends list), because currently it does the res.render before it finishes the loop.
Code:
function onlineFriends(req, res) {
var onlinefriends = new Array();
onlinefriends.push("mark");
FriendList.findOne({
owner: req.session.username
}, function (err, friendlist) {
friendlist.friends.forEach(function (friend) { // here forEach starts
OnlineUser.findOne({
userName: friend
}, function (err, onlineFriend) {
if (onlineFriend != null) {
onlinefriends.push(onlineFriend.userName);
console.log("a loop");
}
});
});
console.log("online friends: " + onlinefriends);
console.log("redirecting");
res.render('index', { // this is still inside the forEach function
friendlist: friendlist.friends,
onlinefriendlist: onlinefriends,
username: req.session.username
});// and here it ends
});
}
output will be as follows:
online friends: mark
redirecting
a loop
a loop
a loop
a loop
a loop
a loop
a loop
As discussed here ( JavaScript, Node.js: is Array.forEach asynchronous? ) , the answer is that the for-each is blocking, but in my example it seems to be non-blocking because it executes the res.render before it has finished looping?
How can I make sure that the for each is finished so I have an up to date onlinefriends list (and friendlist) which I can than pass to the res.render instead of the res.render happening way before the for -each loop finishes (which gives me an incorrect list of online users) ?
Thanks very much!
The following console log:
console.log("a loop");
is inside a callback
I believe that the callback of the function OnlineUser.findOne() is called asynchronously, that is why the code will log "a loop" after the redirect log
You should put the redirection after all the loop callbacks have been executed
Something like:
var count = 0;
friendlist.friends.forEach(function (friend) { // here forEach starts
OnlineUser.findOne({
userName: friend
}, function (err, onlineFriend) {
count++;
if (onlineFriend != null) {
onlinefriends.push(onlineFriend.userName);
console.log("a loop");
}
if(count == friendlist.friends.length) { // check if all callbacks have been called
redirect();
}
});
});
function redirect() {
console.log("online friends: " + onlinefriends);
console.log("redirecting");
res.render('index', { // this is still inside the forEach function
friendlist: friendlist.friends,
onlinefriendlist: onlinefriends,
username: req.session.username
});// and here it ends
}
I was able to solve something similar by adding the async package to my project and changing forEach() to async.each(). The advantage is that this provides a standard way to do synchronization for other parts of the application.
Something like this for your project:
function onlineFriends(req, res) {
var onlinefriends = new Array();
onlinefriends.push("mark");
FriendList.findOne({owner: req.session.username}, function (err, friendlist) {
async.each(friendlist.friends, function(friend, callback) {
OnlineUser.findOne({userName: friend}, function (err, onlineFriend) {
if (onlineFriend != null) {
onlinefriends.push(onlineFriend.userName);
console.log("a loop");
}
callback();
});
}, function(err) {
console.log("online friends: " + onlinefriends);
console.log("redirecting");
res.render('index', { // this is still inside the forEach function
friendlist: friendlist.friends,
onlinefriendlist: onlinefriends,
username: req.session.username
});
});
});
}
Running your code through jsbeautifier indents it properly and shows you why that happens:
function onlineFriends(req, res) {
var onlinefriends = new Array();
onlinefriends.push("mark");
FriendList.findOne({
owner: req.session.username
}, function (err, friendlist) {
friendlist.friends.forEach(function (friend) { // here forEach starts
console.log("vriend: " + friend);
OnlineUser.findOne({
userName: friend
}, function (err, onlineFriend) {
if (onlineFriend != null) {
onlinefriends.push(onlineFriend.userName);
console.log("online friends: " + onlinefriends);
}
});
console.log("nu door verwijzen");
res.render('index', { // this is still inside the forEach function
friendlist: friendlist.friends,
onlinefriendlist: onlinefriends,
username: req.session.username
});
}); // and here it ends
});
So... always indent your code properly and you won't have issues like this. Some editors such as Vim can indent your whole file with a single shortcut (gg=G in vim).
However, OnlineUser.findOne() is most likely asynchronous. so even if you move the call to the correct location it won't work. See ShadowCloud's answer on how to solve this.
Related
I did a couple of projects with node.js and I'm aware of the async behaviour and that one should usually use callback functions, etc. But one thing that bothers me ist the following.
I'm developing an Alexa skill and I have a function that handles the User intent:
'MyFunction': function() {
var toSay = ""; // Holds info what Alexa says
// Lot of checks and calculations what needs to be said by Alexa (nothing special)
if(xyz) {
toSay = "XYZ";
}else if(abc) {
toSay = "ABC";
}else{
toSay = "Something";
}
// Here is the "tricky" party
if(someSpecialEvent) {
toSay += " "+askDatabaseForInput(); // Add some information from database to string
}
this.emit(':ask', toSay, this.t('REPROMT_SPEECH')); // Gives the Info to Alexa (code execution stops here)
}
As mentioned in the code, there is some code which is usually used to find out what the output to Alexa should be.
Only on rare events, "someSpecialEvent", I need to query the database and add information to the String "toSay".
Querying the DB would look something like:
function askDatabaseForInput() { // The function to query the DB
var params = {
TableName: "MyTable",
OtherValues: "..."
};
// Do the Query
docClient.query(params, function(err, data) {
// Of course here are some checks if everything worked, etc.
var item = data.Items[0];
return item; // Item SHOULD be returned
});
return infoFromDocClient; // Which is, of course not possible
}
Now I know, that in the first function "'MyFunction'" I could just pass the variable "toSay" down to the DB Function and then to the DB Query and if everything is fine, I would do the "this.emit()" in the DB Query function. But for me, this looks very dirty and not much reusable.
So is there a way I can use "askDatabaseForInput()" to return DB information and just add it to a String? This means making the asynchronous call synchronous.
Making a synchronous call wouldn't affect the user experience, as the code isn't doing anything else anyway and it just creates the String and is (maybe) waiting for DB input.
Thanks for any help.
So you could do 2 things:
Like the person who commented says you could use a callback:
function askDatabaseForInput(callback) {
var params = {
TableName: "MyTable",
OtherValues: "..."
};
docClient.query(params, function(err, data) {
if (err) {
callback(err, null)
} else {
var item = data.Items[0];
callback(null, item);
}
});
}
or you could use promises:
function askDatabaseForInput() {
var params = {
TableName: "MyTable",
OtherValues: "..."
};
return new Promise(function (resolve, reject) {
docClient.query(params, function(err, data) {
if (err) {
reject(err)
} else {
var item = data.Items[0];
resolve(item);
}
});
});
}
you can then either put a function in where you call askDatabaseForInput or do askDatabaseForInput.then(....).
In the function or the .then you would add what you retrieved from the database to the variable toSay
hope this helps
I'm using Node.js with MongoDB, I'm also using Monk for db access. I have the below code :
console.time("start");
collection.findOne({name: "jason"},
function(err, document) {
for(var i = 0; i < document.friends.length; i++) // "friends is an array contains ids of the user's friends"
{
collection.findOne({id: document.friends[i]}, function(err, doc)
{
console.log(doc.name);
});
}
});
console.log("The file was saved!");
console.timeEnd("start");
I have two questions regarding this code :
I see the execution time and "The file was saved!" string first, then I see the names of the friends coming in the console. Why is that? Shouldn't I see the names first then the execution time? Is it because the async nature of Node.js?
Names are printing very slowly in the console, the speed is like one name in two seconds. Why is it so slow? Is there a way to make the process faster?
EDIT:
Is it a good idea to break friends list to smaller pieces and call friends asynchronously? Would it make the process faster?
EDIT 2:
I changed my code to this :
collection.find({ id: { "$in": document.friends}}).then(function(err, doc)
{
console.log(doc.name);
if(err) {
return console.log(err);
}
}
This doesn't give an error, but this doesn't print anything either.
Thanks in advance.
Answer for question 1:
Yes, you are right.
Is it because the async nature of Node.js.
And to prevent that Node.js provides some mechanism for that you can use it otherwise you can do it on your own manually by setting one flag.
Answer for question 2:
you can use $in instead of findOne, it will be ease and fast.
e.g. .find({ "fieldx": { "$in": arr } })
arr :- In this you need to provide whole array.
yes, it's because javascript's async nature.
As you have called db from for loop javascript will not wait for it's response and continue the execution so it will print the file was saved first.
about your ans 2
It's making a dbCall for every friend then it's obvious that it will take some time that's why it's taking 1 or 2 secs for every friend.
console.time("start");
collection.findOne({name: "jason"},
function(err, document) {
for(var i = 0; i < document.friends.length; i++) // "friends is an array contains ids of the user's friends"
{
console.log("InsideforLoop Calling " + i + " friend");
collection.findOne({id: document.friends[i]}, function(err, doc)
{
console.log(doc.name);
});
console.log("Terminating " + i + "-----");
}
});
console.log("The file was saved!");
console.timeEnd("start");
This will make your async and db doubts more clear.
As you will see it will print all console in line.
InsideforLoop Calling 0 friend
Terminating 0 -----
and so on....Like this
console.log(doc.name);
but this will be printed asynchronusly
Added
collection.findOne({name: "jason"},
function(err, document) {
//you can do this
collection.find({id: $in:{document.friends}, function(err, doc)
{
console.log(doc);
});
});
Find All Details in one call
collection.aggregate([
{
$match:{
id :{ "$in" : document.friends},
}
}
]).exec(function ( e, d ) {
console.log( d )
if(!e){
// your code when got data successfully
}else{
// your code when you got the error
}
});
collection.findOne({name: "jason"},
function(err, document) {
if(document != undefined){
collection.find({ id: { "$in": document.friends}}).then(function(err, doc)
{
console.log(doc.name);
if(err) {
return console.log(err);
}
}
}
});
Answer to 1: Yes, it is because node is async. The part where it logs names is executed only when the first findOne returns, whereas the file was saved is executed straight away.
Ive written and basic Node app (my first) to insert many csv rows into mongo (items array in the code below). Once all items have been inserted the db connection should be closed and the program exited.
The issue ive been working with is figuring out when to close the db connection once all inserts have returned a result. Ive gotten it working by counting all of the insert result callbacks but to me this feels clunky. I know one improvement I could make is to batch the inserts via an array to the insert function but ill still need to have my code be aware of when all inserts have completed (assuming it would be bad to insert 100k items in one query). Is there and better way (my code feels hacky) to do this?
Hack part...
function (err, result) {
queryCompletedCount++;
if (err) console.log(err);
//Not sure about doing it this way
//Close db once all queries have returned a result
if (queryCompletedCount === items.length) {
db.close();
console.log("Finish inserting data: " + new Date());
}
}
Full insert code
MongoClient.connect(dbConnectionURL, function (err, db) {
if (err) {
console.log("Error connecting to DB: " + err);
} else {
var productCollection = db.collection('products');
console.log("Connected to DB");
console.log("Start inserting data: " + new Date());
var queryCompletedCount = 0;
for (var i = 0; i < items.length; i++) {
productCollection.insert([{
manufacturerCode: null,
name: items[i].name,
description: null
}], function (err, result) {
queryCompletedCount++;
if (err) console.log(err);
//Not sure about doing it this way
//Close db once all queries have returned a result
if (queryCompletedCount === items.length) {
db.close();
console.log("Finish inserting data: " + new Date());
}
});
}
}
});
What do you think about realizing this issue with async module like this:
async = require('async')
async.eachSeries(items, function (item, next) {
productCollection.insert(productCollection.insert(
[{
manufacturerCode: null,
name: item.name,
description: null
}], function (err, result) {
if (err) {
return next(err);
}
next();
})
)
}, function () {
// this will be called after all insertion completed
db.close();
console.log("Finish inserting data: " + new Date());
});
What you need here is MongoDB's Write Concern, configured in the strictest way.
There are two levels of Write Concern. The first is the write mode, in which case the query returns only if the result is written to the configured number of mongo instances. In your case I suppose there is a single instance, but for future you may configure it as "w": "majority". The second level is the Journal concern, where by setting "j": 1 your query will return only when the data is written into the journal.
So in your case you best Write Concern configuration might be {"w": "majority", "j": 1}. Just add it as the last argument of your insert statement.
I am literally giving my first steps with node and mongodb and I have recently hit this RangeError wall.
Here's what I am trying to do, I have a file that contains a list of countries that I would like to add to my mongo db. This would be part of my "seed" mechanism to get the app running.
I load the json and then I iterate through the collection of objects and add them one by one to the 'Countries' collection.
However, everytime I run the code, I get a "RangeError: Maximum call stack size exceeded".
I have googled around but none of the suggested solutions seem to apply for me.
My guess is there is something wrong with my insertCountry function...
Anyways, here's my code:
var mongoose = require('mongoose');
var countries = require('./seed/countries.json');
// mongodb
var Country = mongoose.Schema({
name: String,
code: String,
extra: [Extra]
});
var Extra = mongoose.Schema({
exampleField: Boolean,
anotherField: Boolean
});
var mCountry = mongoose.model('Countries', Country);
var mExtra = mongoose.model('Extras', Extra);
// do connection
mongoose.connect('...');
var db = mongoose.connection;
db.on('error', console.error.bind(console, 'connection error'));
db.once('open', function callback() {
});
// async function
var insertCountry = function(document, callback) {
db.model('Countries').count({code: document.code}, function (err, count) {
if (count < 1) {
db.collection('Countries').insert(document, function (err, result) {
if (!err) {
console.log('country ' + document.name + ' added');
}
else {
console.log('- [' + document.name + '] ' + err);
}
});
}
callback(null,document);
});
};
// doing countries
var Country = mongoose.model('Countries');
var Extras = mongoose.model('Extras');
for(i = 0; i < countries.length; i++)
{
nCountry = new Country();
nCountry.name = countries[i].name;
nCountry.code = countries[i].code;
nCountry.benefits = new Extras();
nCountry.benefits.exampleField = false;
nCountry.benefits.anotherField = false;
insertCountry(nCountry, function (err, value) {
console.log(value.name + ' added to collection (callback)');
});
}
I have been using some guides I have found to build this so this might not be optimal code. Any best pratices, standards, guides or tutorials you can share are most welcome!
Your callback is in the wrong place. It is not waiting for the insert operation to complete before you return from it's own callback. Altering your code:
var insertCountry = function(document, callback) {
db.model('Countries').count({code: document.code}, function (err, count) {
if (count < 1) {
db.collection('Countries').insert(document, function (err, result) {
if (!err) {
console.log('country ' + document.name + ' added');
}
else {
console.log('- [' + document.name + '] ' + err);
}
callback(null,document);
});
}
});
};
That is part of your problem, but it does not completely solve it. The other part is the loop which also does not wait for the wrapping function to complete before moving on. You want something like asyc.eachSeries in order to wait for inserts to complete before performing the next iteration. This is mostly why you are exceeding the call stack:
async.eachSeries(
countries,
function(current,callback) {
// make your nCountry object
insertCountry(nCountry,function(err,value) {
// do something, then
callback(err);
})
},
function(err) {
// called where done, err contains err where set
console.log( "done" );
}
);
There is really still and issue with the array, which must be reasonably large if you are exceeding the call stack limit. You probably should look at using event streams to process that rather that load everything in memory to the array.
Personally, if you were just trying not to insert duplicates for a field and had MongoDB 2.6 available I would just use the Bulk Operations API with "unordered operations" and allow non fatal failures on the duplicate keys. Coupled with the fact that bulk operations are sent in "batches" and not one at a time, this is much more efficient than checking for the presence on every request:
var Country = mongoose.Schema({
name: String,
code: { type: String, unique: true }, // define a unique index
extra: [Extra]
});
var insertCountries = function(countries,callback) {
var bulk = Country.collection.initializeUnorderedBulkOp();
var counter = 0;
async.eachSeries(
countries,
function(current,callback) {
// same object construction
bulk.insert(nCountry);
counter++;
// only send once every 1000
if ( counter % 1000 == 0 ) {
bulk.execute(function(err,result) {
// err should generally not be set
// but result would contain any duplicate errors
// along with other insert responses
// clear to result and callback
bulk = Country.collection.initializeUnorderedBulkOp();
callback();
});
} else {
callback();
}
},
function(err) {
// send anything still queued
if ( counter % 1000 != 0 )
bulk.execute(function(err,result) {
// same as before but no need to reset
callback(err);
});
}
);
};
mongoose.on("open",function(err,conn) {
insertCountries(countries,function(err) {
console.log("done");
});
});
Keeping in mind that unlike the methods implemented directly on the mongoose models, the native driver methods require that a connection is actually established before they can be called. Mongoose "queues" these up for you, but otherwise you need something to be sure the connection is actually open. The example of the "open" event is used here.
Take a look at event streams as well. If you are constructing an array large enough to cause a problem by missing callback execution then you probably should not be loading it all in memory from whatever your source is. Stream processing that source combined with an approach as shown above should provide efficient loading.
I'm writing a multiplayer game(mongojs, nodejs) and trying to figure out how to update user stats based on the outcome of the game. I already have the code written to compute all the post game stats. The problem comes when I try to update the users' stats in a for loop. Here's what i got:
//Game Stats
var tempgame = {
gameid: 1234,
stats: [
{
score: 25,
user: 'user1'
},
{
score: 25,
user: 'user2'
}
]
}
for(i = 0; i < tempgame.stats.length; i++){
db.users.find({ username: tempgame.stats[i].user }, function(err, res){
if( err != null){
//handle errors here.
} else {
var userstats = res[0].stats;
if( tempgame.stats[i].score > userstats.bestscore ){ //this is where it chokes
userstats.bestscore = tempgame.stats[i].score;
}
//code here to pass back new manipulated stats
}
});
}
Everything works fine until i try to use the tempgame object within the callback function. It says "cannot read property 'score' of undefined". Is this just a scoping issue?
Also i was thinking it could be an issue with the callback function itself. Maybe the loop would increment before the callback is even run. But even in that case, the score should be be there it would just be pulling from the wrong array index... that's what lead me to believe it may just be a scope issue.
Any help would be greatly appreciated.
You've been tripped up by the notorious "defining functions inside a loop" problem.
Use "forEach" instead:
tempgame.stats.forEach(function (stat) {
db.users.find({ username: stat.user }, function(err, res){
if( err != null){
//handle errors here.
} else {
var userstats = res[0].stats;
if( stat.score > userstats.bestscore ){ //this is where it chokes
userstats.bestscore = stat.score;
}
//code here to pass back new manipulated stats
}
});
});
Part of your problem is as mjhm stated in his answer to your question, and is as you have suspected. The i variable is changing before the callback is invoked.
The other half of your problem is because your database calls have not returned yet. Due to the asynchronous nature of NodeJS, your loop will finish before your database calls complete. Additionally, your database calls are not necessarily coming back in the same order you called them. What you need is some sort of flow control like async.js. Using async.map will allow you to make all calls to the DB in parallel and return them as an array of values you can use, after all db calls have been completed.
async.map(tempgame.stats, function(stat, callback){
db.users.find({ username: stat.user }, function(err, res){
if( err != null){
callback(err);
} else {
callback(null, res[0].stats);
}
});
}, function(err, stats){
if(err){
//handle errors
} else{
stats.forEach(function(stat){
//do something with your array of stats
//this wont be called until all database calls have been completed
});
}
});
In addition to the above, if you want to return results back to the application,
http://nodeblog.tumblr.com/post/60922749945/nodejs-async-db-query-inside-for-loop