We Keep Coding

Loopback: Querying multiple collections not asynchronously

I have a MongoDB database with 2 collections: Customer and Order. These models' relationships in Loopback are as follows: Customer -hasMany-> Order & Order -belongsTo-> Customer (so an order has a foreignKey customerId).
I needed to query the orders of all customers that have age 20 for example. But I was surprised to notice that Loopback doesn't support this kind of queries. I searched a lot and even the "include" option doesn't support the format I want to get as a final result and cannot be combined to the filter by age functionality (Loopback-doc-about-it). Then, I wrote a remote method that makes 2 queries: the first one finds all customers who have a certain age (Basic where filter) and the other iterates over every customer in that list to find his orders ( basically for each index, it searches in the Orders collection which ones has customerId equal to the index's customer's id)
Here's customer.js file:
'use strict';
module.exports = function(Customer) {
var app = require('../../server/server');
/**
*
* #param {number} age
* #param {Function(Error, array)} callback
*/
Customer.getOrdersByAge= function(age, callback) {
var customers;
var filter= { where: { 'age': age } };
var Order=app.models.Order;
var orders;
var elementary_orders;
Customer.find(filter, function(err, items) {
if (err !==null){
console.log("error1");
return callback(err);
}
console.log("items: "+ items);
customers=items;
for (let i of customers){
console.log(i+": "+i.id+" -lenght: "+customers.length);
var filter_order= { where: { 'customerId': i.id+'' } };
Order.find(filter_order, function(err2, items_orders) {
if (err2 !==null){
console.log(i+": "+"error2");
return callback(err2);
}
elementary_orders= items_orders;
orders=elementary_orders;
console.log("elementary_orders: "+ elementary_orders);
console.log("orders now: "+ elementary_orders);
});
}
console.log("-> orderssss: "+ orders);
callback(null,orders);
});
}
};
But then I found another problem, orders is always undefined. It appears that since "find" queries are asynchronous, orders stays undefined. So, I looked for a way to make it synchronous (but it was impossible since it's Loopback's spirit to be always asynchronous) and a way to control the flow through npm's (async package but even after trying the eachOf utility, orders is still undefined.
I am wondering not only how can I make this simple query work but also why is it so impossible to implement? Am I violating any conceptual or architectural patterns related to Loopback's models or something? Querying multiple collections is a usual thing to do, though.
Thank you :)

You can use async.map on the customers to patch the models and get the orders added to them.
async.map(customers, function(customer, mapCallback) {
Order.find(params, function (orders, error) {
if(ordersError) {
mapCallback(null, ordersError);
}
customer.orders = orders;
mapCallback(customer);
});
}, callback); //This callback is the main from the remote method.

Will Mongo handle my service?

I've been using MongoDB with node.js and mongoose library. I decided to start using MongoDB because I found everywhere that it is the best solution for node.js applications.
Although the response times of my API are good, I'm unsure that MongoDB will handle it when scaling it.
I've noticed that most of my queries aren't enough to get all the data I need, so I rely on creating several queries and using some javascript map/reduce functions (that is what I'm afraid of).
Look at this example:
User
.find({
idol : true
})
.sort({
'metas.followers' : -1
})
.select('-password -__v -posts -email')
.skip(offset)
.limit(30)
.exec(function(err, retData)
{
promisedIdols = retData.map(function(idol)
{
return idol.withStatistics(Post, Follow, req.user);
});
idols = [];
if(promisedIdols.length == 0)
{
callback();
}
for(var i=0; i<promisedIdols.length; i++)
{
promisedIdols[i].delegate(function(result)
{
idols.push(result);
if(idols.length == promisedIdols.length)
{
callback();
}
});
}
});
I've used a map to gather an array of promises that will be resolved after running the following code:
var obj = this.toObject();
var deferred = new Promise();
Post
.find({ idol : obj._id })
.lean()
.exec(function(err, posts)
{
var postViews = 0;
var postLikes = 0;
var postShares = 0;
posts.reduce(function(prev, next)
{
postViews += next.views.length;
postLikes += next.likes.length;
postShares += next.shares.length;
}, 0);
obj.metas.postViews = postViews;
obj.metas.postLikes = postLikes;
obj.metas.postShares = postShares;
obj.metas.postCount = posts.length;
Subscription
.count({ idol : obj._id }, function(err, count)
{
obj.metas.subscribers = count;
deferred.deliver(obj);
});
});
that uses a reduce function.
I can't see this code working well on big scale. Maybe should I restructure my database? Maybe should I change my database system? Maybe I'm using MongoDB wrongly?
Experts?
Thanks.

Mongo can handle a lot, if you setup a good data model. There are a few things to keep in mind when you want to scale.
Try to avoid normalizing the data much and split it into different collections.
Data duplication is (sometimes, when used wisely) your friend, it will help you make simpler queries, populate right away. Yeah, that may mean that when you're updating data, you'll have to update in two places, but Mongo is ok with a lot of writes if you do it asynchronously (promises or not).
To your specific query, I don't see the full data model, but maybe you can use aggregation framework. That pipeline is native (C++, as opossed to mapReduce JavaScript) and will work really really fast.
Something like:
db.post.aggregate(
// First $match to reduce the dataset
{
$match: {idol : obj._id}
},
// then group and aggregate your data
{
$group: {
_id: '$idol', // group by that idol thing
postViews: {$sum: '$postViews'},
postLikes: {$sum: '$postLikes'}
},
},
// Then use project to arrange the result the way you like it
{
$project: {
_id: false, //or true if you need it
metas: {
postViews: '$postViews'
},
likeCountOfPosts: '$postLikes', // that's how you'd rename
whatIsIt: {$literal: 'a great post'}
}
}
);
You can also do a lot of conditional, groupings, sortings, winding and unwinding, mixing and shuffling the pipeline.
It's much much faster then Mongo mapReduce.

Problems with facade in javascript and Mongo

I am writting a little app to calculate and keep track of how much gas my car is using. But i have a problem with my facade.
When i am trying to add some new details to my database, i only get a empty object somehow.
I know the problem is in my facade, and maybe some of you can see what it is.
(Don't worry about the name of the method)
function addTitle (kilometer, liter, kmLiter, callback){
var data = {
kilometer: kilometer,
liter: liter,
kmLiter: kmLiter
}
detail.create({details:data}, function(err, result){
if(err)
return callback(err);
else
callback(null, result);
});
};
And this is the model of the DB
var DetailSchema = mongoose.Schema({
details:[{
kilometer: String,
liter: String,
kmLiter: String}
]
});
mongoose.model('Details', DetailSchema, "details");
Anybody that can find the error?

Assuming your schema definition of details being an array is what you want, you need to also make details an array when you create new docs.
So your function should change to:
function addTitle (kilometer, liter, kmLiter, callback){
var data = {
kilometer: kilometer,
liter: liter,
kmLiter: kmLiter
}
detail.create({details: [data]}, callback);
}

Best method to string together variety of DB calls in Node js

I basically need to make about 3 calls to get the data for a json object.. It basically JSON array of JSON object which have some attributes, one of which is an array of other values selected using a second query, then that one also has an array inside which is selected with another db call.
I tried using asyn.concatSeries so that I can dig down into the bottom call and put together all the information I collected for one root json object but that's creating a lot of unexpected behaviour..
Example of JSON
[
{
"item" : "firstDbCall"
"children" : [ {
"name" : "itemDiscoveredWithSecondDBCall"
"children" : [ itemsDiscoveredwith3rdDBCall]
},
]
}
]
This is really difficult using node.js. I really need to figure out how to do this properly since I have to do many of these for different purposes.
EDIT
This is the code i have. There's some strange behaviour with async.concatSeries. The results get called multiple times after each one of the functions finish for each array. So i had to put a check in place. I know it's very messy code but i've been just putting band-aids all over it for the past 2 hours to make it work..
console.log("GET USERS HAREDQARE INFO _--__--_-_-_-_-_____");
var query = "select driveGroupId from tasks, driveInformation where agentId = '"
+ req.params.agentId + "' and driveInformation.taskId = tasks.id order by driveInformation.taskId desc;";
connection.query(query, function(err, rows) {
if (rows === undefined) {
res.json([]);
return;
}
if(rows.length<1) { res.send("[]"); return;}
var driveGroupId = rows[0].driveGroupId;
var physicalQuery = "select * from drives where driveGroupId = " + driveGroupId + ";";
connection.query(physicalQuery, function(err, rows) {
console.log("ROWSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS");
console.log(rows);
async.concatSeries(rows, function(row, cb) {
console.log("-------------------------------SINGLE ROW-------------------------------------");
console.log(row);
if(row.hasLogicalDrives != 0) {
console.log("HAS LOGICAL DRIVES");
console.log(row.id);
var query = "select id, name from logicalDrives where driveId = " + row.id;
connection.query(query, function(error, drives) {
console.log("QUERY RETURNED");
console.log(drives);
parseDriveInfo(row.name, row.searchable, drives, cb);
});
}
else
var driveInfo = { "driveName" : row.name, "searchable" : row.searchable};
console.log("NO SUB ITEMS");
cb(null, driveInfo);
}, function(err, results) {
console.log("GEETTTTINGHERE");
console.log(results);
if(results.length == rows.length) {
console.log("RESULTS FOR THE DRIVE SEARCH");
console.log(results);
var response = {"id": req.params.agentId};
response.driveList = results;
console.log("RESPONSE");
console.log(response);
res.json(response);
}
});
});
});
};
parseDriveInfo = function(driveName, searchable, drives, cb) {
async.concatSeries(drives, function(drive,callback) {
console.log("SERIES 2");
console.log(drive);
console.log("END OF DRIVE INFO");
var query = "select name from supportedSearchTypes where logicalDriveId = " + drive.id;
connection.query(query, function(error, searchTypes) {
drive.searchTypes = searchTypes;
var driveInfo = { "driveName" :driveName,
"searchable" : searchable,
"logicalDrives" : drive
};
callback(null, driveInfo);
});
}, function (err, results) {
console.log("THIS IS ISISIS ISISISSISISISISISISISISISIS");
console.log(results);
if(results.length === drives.length) {
console.log("GOTHERE");
cb(null, results);
}
});
}

Getting good enough with async to use exactly the right combination of methods under the right circumstances takes a fair amount of experience. Most likely your case in particular can be handled with async.waterfall if its query1 then query2(dataFoundByQuery1) then query3(dataFoundByQuery2). But depending on the circumstances you need to mix and match async methods appropriately and sometimes have 2 levels - for example a "big picture" async.waterfall where some of the steps in the waterfall do async.parallel or async.series as needed. I've never used async.concat and given your needs I think you have chosen the wrong method. The workhorses are async.each, async.eachSeries, async.waterfall, and async.map, at least for the web app & DB query use cases I mostly encounter, so make sure you really have those understood before exploring the more specific convenience methods.

EDIT: This is a more in depth example based on use of the connection library you seem to be using. Please note, some of this is javascript psuedo code. Things like adding objects to the resultsArray are clearly not complete, the only thing I took time to make sure was correct is the "flow of logic" as it pertains to callbacks. Everything else is for you to implement. In order to support multiple calls to the same callback function and maintain state from call to call, the best way is to wrap the set of callbacks in a closure. This allows the callbacks to share some state with the main event loop. This allows you to pass arguments to the callbacks, without actually having to pass them as arguments, much like class variables in c++, or even globals in javascript, but we haven't poluted the global scope :)
function queryDataBase(query) {
//wrap the whole query in a function so the callbacks can share some
//variables with similar scope. This is called a closure
int rowCounter = 0;
var dataRowsFromStep2;
var resultsArray = {};
connection.query(query, dataBaseQueryStep2);
function dataBaseQueryStep2(err, rows) {
//do something with err and rows
dataRowsFromStep2 = rows;
var query = getQueryFromRow(dataRowsFromStep2[rowCounter++]);//Always zero the first time. Might need to double check rows isn't empty!
connection.query(query, dataBaseQueryStep3);
}
function dataBaseQueryStep3(err, rows) {
//do something with err and rows
if(rowCounter < dataRowsFromStep2.size) {
resultsArray.add(rows);//Probably needs to be more interesting, but you get the idea
//since this is within the same closure, rowCounter maintains it's state
var query = getQueryFromRow(dataRowsFromStep2[rowCounter++]);
//recursive call query using dataBaseQueryStep3 as it's callback repeatedly until
//we run out of rows to call it on.
connection.query(query, dataBaseQueryStep3)
} else {
//when the if statement fails we have no more rows to run queries on so return to main program flow
returnToMainProgramLogic(resultsArray);
}
}
}
function returnToMainProgramLogic(results) {
//continue running your program here
}
I personally like the above logic better than the syntax async produces... I believe the heart of your problem rests in your nested calls to async, and the fact that ASYN itself, runs the series of functions asynchronously, but in order(confusing I know). If you write your program like this, you won't have to worry about it!

I would strongly suggest using sequelize.js It provides a really powerful orm that allows you to chain queries together. It also allows you to directly load your data into js objects, write dynamic sql, and connect to many different databases. Picture ActiveRecord from the Ruby world for Node.

How to use "q" module for refactoring mongoose code?

I'm using mongoose to insert some data into mongodb. The code looks like:
var mongoose = require('mongoose');
mongoose.connect('mongo://localhost/test');
var conn = mongoose.connection;
// insert users
conn.collection('users').insert([{/*user1*/},{/*user2*/}], function(err, docs) {
var user1 = docs[0], user2 = docs[1];
// insert channels
conn.collection('channels').insert([{userId:user1._id},{userId:user2._id}], function(err, docs) {
var channel1 = docs[0], channel2 = docs[1];
// insert articles
conn.collection('articles').insert([{userId:user1._id,channelId:channel1._id},{}], function(err, docs) {
var article1 = docs[0], article2 = docs[1];
}
});
};
You can see there are a lot of nested callbacks there, so I'm trying to use q to refactor it.
I hope the code will look like:
Q.fcall(step1)
.then(step2)
.then(step3)
.then(step4)
.then(function (value4) {
// Do something with value4
}, function (error) {
// Handle any error from step1 through step4
})
.end();
But I don't know how to do it.

You'll want to use Q.nfcall, documented in the README and the Wiki. All Mongoose methods are Node-style. I'll also use .spread instead of manually destructuring .then.
var mongoose = require('mongoose');
mongoose.connect('mongo://localhost/test');
var conn = mongoose.connection;
var users = conn.collection('users');
var channels = conn.collection('channels');
var articles = conn.collection('articles');
function getInsertedArticles() {
return Q.nfcall(users.insert.bind(users), [{/*user1*/},{/*user2*/}]).spread(function (user1, user2) {
return Q.nfcall(channels.insert.bind(channels), [{userId:user1._id},{userId:user2._id}]).spread(function (channel1, channel2) {
return Q.nfcall(articles.insert.bind(articles), [{userId:user1._id,channelId:channel1._id},{}]);
});
})
}
getInsertedArticles()
.spread(function (article1, article2) {
// you only get here if all three of the above steps succeeded
})
.fail(function (error) {
// you get here if any of the above three steps failed
}
);
In practice, you will rarely want to use .spread, since you usually are inserting an array that you don't know the size of. In that case the code can look more like this (here I also illustrate Q.nbind).
To compare with the original one is not quite fair, because your original has no error handling. A corrected Node-style version of the original would be like so:
var mongoose = require('mongoose');
mongoose.connect('mongo://localhost/test');
var conn = mongoose.connection;
function getInsertedArticles(cb) {
// insert users
conn.collection('users').insert([{/*user1*/},{/*user2*/}], function(err, docs) {
if (err) {
cb(err);
return;
}
var user1 = docs[0], user2 = docs[1];
// insert channels
conn.collection('channels').insert([{userId:user1._id},{userId:user2._id}], function(err, docs) {
if (err) {
cb(err);
return;
}
var channel1 = docs[0], channel2 = docs[1];
// insert articles
conn.collection('articles').insert([{userId:user1._id,channelId:channel1._id},{}], function(err, docs) {
if (err) {
cb(err);
return;
}
var article1 = docs[0], article2 = docs[1];
cb(null, [article1, article2]);
}
});
};
}
getInsertedArticles(function (err, articles) {
if (err) {
// you get here if any of the three steps failed.
// `articles` is `undefined`.
} else {
// you get here if all three succeeded.
// `err` is null.
}
});

With alternative deferred promise implementation, you may do it as following:
var mongoose = require('mongoose');
mongoose.connect('mongo://localhost/test');
var conn = mongoose.connection;
// Setup 'pinsert', promise version of 'insert' method
var promisify = require('deferred').promisify
mongoose.Collection.prototype.pinsert = promisify(mongoose.Collection.prototype.insert);
var user1, user2;
// insert users
conn.collection('users').pinsert([{/*user1*/},{/*user2*/}])
// insert channels
.then(function (users) {
user1 = users[0]; user2 = users[1];
return conn.collection('channels').pinsert([{userId:user1._id},{userId:user2._id}]);
})
// insert articles
.match(function (channel1, channel2) {
return conn.collection('articles').pinsert([{userId:user1._id,channelId:channel1._id},{}]);
})
.done(function (articles) {
// Do something with articles
}, function (err) {
// Handle any error that might have occurred on the way
});

Considering Model.save instead of Collection.insert (quite the same in our case).
You don't need to use Q, you can wrap yourself the save method and return directly a Mongoose Promise.
First create an utility method to wrap the save function, that's not very clean but something like:
//Utility function (put it in a better place)
var saveInPromise = function (model) {
var promise = new mongoose.Promise();
model.save(function (err, result) {
promise.resolve(err, result);
});
return promise;
}
Then you can use it instead of save to chain your promises
var User = mongoose.model('User');
var Channel = mongoose.model('Channel');
var Article = mongoose.model('Article');
//Step 1
var user = new User({data: 'value'});
saveInPromise(user).then(function () {
//Step 2
var channel = new Channel({user: user.id})
return saveInPromise(channel);
}).then(function (channel) {
//Step 3
var article = new Article({channel: channel.id})
return saveInPromise(article);
}, function (err) {
//A single place to handle your errors
});
I guess that's the kind of simplicity we are looking for.. right? Of course the utility function can be implemented with better integration with Mongoose.
Let me know what you think about that.
By the way there is an issue about that exact problem in the Mongoose Github:
Add 'promise' return value to model save operation
I hope it's gonna be solved soon. I think it takes some times because they are thinking of switching from mpromise to Q: See here and then here.

Two years later, this question just popped up in my RSS client ...
Things have moved on somewhat since May 2012 and we might choose to solve this one in a different way now. More specifically, the Javascript community has become "reduce-aware" since the decision to include Array.prototype.reduce (and other Array methods) in ECMAScript5. Array.prototype.reduce was always (and still is) available as a polyfill but was little appreciated by many of us at that time. Those who were running ahead of the curve may demur on this point, of course.
The problem posed in the question appears to be formulaic, with rules as follows :
The objects in the array passed as the first param to conn.collection(table).insert() build as follows (where N corresponds to the object's index in an array):
[ {}, ... ]
[ {userId:userN._id}, ... ]
[ {userId:userN._id, channelId:channelN._id}, ... ]
table names (in order) are : users, channels, articles.
the corresopnding object properties are : user, channel, article (ie the table names without the pluralizing 's').
A general pattern from this article by Taoofcode) for making asynchronous call in series is :
function workMyCollection(arr) {
return arr.reduce(function(promise, item) {
return promise.then(function(result) {
return doSomethingAsyncWithResult(item, result);
});
}, q());
}
With quite light adaptation, this pattern can be made to orchestrate the required sequencing :
function cascadeInsert(tables, n) {
/*
/* tables: array of unpluralisd table names
/* n: number of users to insert.
/* returns promise of completion|error
*/
var ids = []; // this outer array is available to the inner functions (to be read and written to).
for(var i=0; i<n; i++) { ids.push({}); } //initialize the ids array with n plain objects.
return tables.reduce(function (promise, t) {
return promise.then(function (docs) {
for(var i=0; i<ids.length; i++) {
if(!docs[i]) throw (new Error(t + ": returned documents list does not match the request"));//or simply `continue;` to be error tolerant (if acceptable server-side).
ids[i][t+'Id'] = docs[i]._id; //progressively add properties to the `ids` objects
}
return insert(ids, t + 's');
});
}, Q());
}
Lastly, here's the promise-returning worker function, insert() :
function insert(ids, t) {
/*
/* ids: array of plain objects with properties as defined by the rules
/* t: table name.
/* returns promise of docs
*/
var dfrd = Q.defer();
conn.collection(t).insert(ids, function(err, docs) {
(err) ? dfrd.reject(err) : dfrd.resolve(docs);
});
return dfrd.promise;
}
Thus, you can specify as parameters passed to cascadeInsert, the actual table/property names and the number of users to insert.
cascadeInsert( ['user', 'channel', 'article'], 2 ).then(function () {
// you get here if everything was successful
}).catch(function (err) {
// you get here if anything failed
});
This works nicely because the tables in the question all have regular plurals (user => users, channel => channels). If any of them was irregular (eg stimulus => stimuli, child => children), then we would need to rethink - (and probably implement a lookup hash). In any case, the adaptation would be fairly trivial.

Today we have mongoose-q as well. A plugin to mongoose that gives you stuff like execQ and saveQ which return Q promises.