I'm attempting to use MongoJS as a wrapper for the native Mongo driver in Node. I'm modeling the documents in my collection as JavaScript classes with methods like populate(), save(), etc.
In most languages like C# and Java, I'm used to explicitly connecting and then disconnecting for every query. Most examples only give an example of connecting, but never closing the connection when done. I'm uncertain if the driver is able to manage this on its own or if I need to manually do so myself. Documentation is sparse.
Here's the relevant code:
User.prototype.populate = function(callback) {
var that = this;
this.db = mongo.connect("DuxDB");
this.db.collection(dbName).findOne({email : that.email}, function(err, doc){
if(!err && doc) {
that.firstName = doc.firstName;
that.lastName = doc.lastName;
that.password = doc.password;
}
if (typeof(callback) === "function"){
callback.call(that);
}
that.db.close();
});
};
I'm finding that as soon as I call the close() method on the MongoJS object, I can no longer open a new connection on subsequent calls. However, if I do not call this method, the Node process never terminates once all async calls finish, as if it is waiting to disconnect from Mongo.
What is the proper way to manage connections to Mongo with MongoJS?
You will get better performance from your application if you leave the connection(s) open, rather than disconnecting. Making a TCP connection, and, in the case of MongoDB, discovering the replica set/sharding configuration where appropriate, is relatively expensive compared to the time spent actually processing queries and updates. It is better to "spend" this time once and keep the connection open rather than constantly re-doing this work.
Don't open + close a connection for every query. Open the connection once, and re-use it.
Do something more like this reusing your db connection for all calls
User = function(db) {
this.db = db;
}
User.prototype.populate = function(callback) {
var that = this;
this.db.collection(dbName).findOne({email : that.email}, function(err, doc){
if(!err && doc) {
that.firstName = doc.firstName;
that.lastName = doc.lastName;
that.password = doc.password;
}
if (typeof(callback) === "function"){
callback.call(that);
}
});
};
I believe it actually closes the connection after each request, but it sets {auto_reconnect:true} in the mongodb server config, so it will reopen a new connection whenever one is needed.
Related
I've been working on a server and a push notification daemon that will both run simultaneously and interact with the same database. The idea behind this is that if one goes down, the other will still function.
I normally use Swift but for this project I'm writing it in Node, using Mongoose as my database. I've created a helper class that I import in both my server.js file and my notifier.js file.
const Mongoose = require('mongoose');
const Device = require('./device'); // This is a Schema
var uri = 'mongodb://localhost/devices';
function Database() {
Mongoose.connect(uri, { useMongoClient: true }, function(err) {
console.log('connected: ' + err);
});
}
Database.prototype.findDevice = function(params, callback) {
Device.findOne(params, function(err, device) {
// etc...
});
};
module.exports = Database;
Then separately from both server.js and notifier.js I create objects and query the database:
const Database = require('./db');
const db = new Database();
db.findDevice(params, function(err, device) {
// Simplified, but I edit and save things back to the database via db
device.token = 'blah';
device.save();
});
Is this safe to do? When working with Swift (and Objective-C) I'm always concerned about making things thread safe. Is this a concern? Should I be worried about race conditions and modifying the same files at the same time?
Also, bonus question: How does Mongoose share a connection between files (or processes?). For example Mongoose.connection.readyState returns the same thing from different files.
The short answer is "safe enough."
The long answer has to do with understanding what sort of consistency guarantees your system needs, how you've configured MongoDB, and whether there's any sharding or replication going on.
For the latter, you'll want to read about atomicity and consistency and perhaps also peek at write concern.
A good way to answer these questions, even when you think you've figured it out, is to test scenarios: Hammer a duplicate of your system with fake data and events and see if what happen is OK or not.
I have a node application handling some ZeroMQ events coming from another application utilizing the Node-ZMQ bindings found here: https://github.com/JustinTulloss/zeromq.node
The issue I am running into is one of the operations from an event takes a long time to process and this appears to be blocking any other event from being processed during this time. Although the application is not currently clustered, doing so would only afford a few more threads and doesn't really solve the issue. I am wondering if there is a way of allowing for these async calls to not block other incoming requests while they process, and how I might go about implementing them.
Here is a highly condensed/contrived code example of what I am doing currently:
var zmq = require('zmq');
var zmqResponder = zmq.socket('rep');
var Client = require('node-rest-client').Client;
var client = new Client();
zmqResponder.on('message', function (msg, data) {
var parsed = JSON.parse(msg);
logging.info('ZMQ Request received: ' + parsed.event);
switch (parsed.event) {
case 'create':
//Typically short running process, not an issue
case 'update':
//Long running process this is the issue
serverRequest().then(function(response){
zmqResponder.send(JSON.stringify(response));
});
}
});
function serverRequest(){
var deferred = Q.defer();
client.get(function (data, response) {
if (response.statusCode !== 200) {
deferred.reject(data.data);
} else {
deferred.resolve(data.data);
}
});
return deferred.promise;
}
EDIT** Here's a gist of the code: https://gist.github.com/battlecow/cd0c2233e9f197ec0049
I think, through the comment thread, I've identified your issue. REQ/REP has a strict synchronous message order guarantee... You must receive-send-receive-send-etc. REQ must start with send and REP must start with receive. So, you're only processing one message at a time because the socket types you've chosen enforce that.
If you were using a different, non-event-driven language, you'd likely get an error telling you what you'd done wrong when you tried to send or receive twice in a row, but node lets you do it and just queues the subsequent messages until it's their turn in the message order.
You want to change REQ/REP to DEALER/ROUTER and it'll work the way you expect. You'll have to change your logic slightly for the ROUTER socket to get it to send appropriately, but everything else should work the same.
Rough example code, using the relevant portions of the posted gist:
var zmqResponder = zmq.socket('router');
zmqResponder.on('message', function (msg, data) {
var peer_id = msg[0];
var parsed = JSON.parse(msg[1]);
switch (parsed.event) {
case 'create':
// build parsedResponse, then...
zmqResponder.send([peer_id, JSON.stringify(parsedResponse)]);
break;
}
});
zmqResponder.bind('tcp://*:5668', function (err) {
if (err) {
logging.error(err);
} else {
logging.info("ZMQ awaiting orders on port 5668");
}
});
... you need to grab the peer_id (or whatever you want to call it, in ZMQ nomenclature it's the socket ID of the socket you're sending from, think of it as an "address" of sorts) from the first frame of the message you receive, and then use send it as the first frame of the message you send back.
By the way, I just noticed in your gist you are both connect()-ing and bind()-ing on the same socket (zmq.js lines 52 & 143, respectively). Don't do that. Inferring from other clues, you just want to bind() on this side of the process.
I am currently building an api application that checks the status and gets information of various types of dbs(i.e. Mongo, MySQL) using Sailsjs such as users, depending on a user input. Here is a snippet of the code I am working on. The local host is just the test database I am connecting to, but in the future it will be supplied by the user.
var mp = require('mongodb-promise');
var MongoClient = require('mongodb');
mp.MongoClient.connect("mongodb://#localhost:27017/test")
.then(function(db){
db.getUsers().then(function(users){
res.ok(users);
})
})
.fail(function(err) {
console.log(err);
})
I am attempting to use promises for the async issue. The problem I am having is that it doesn't work. It tells me that that Object[object object] has no method 'getUsers'. I have searched and can't seem to find a solution that works.
If I change the function to the below, I get the some data back.
mp.MongoClient.connect("mongodb://#localhost:27017/IMS")
.then(function(db){
db.stats().then(function(stats){
return res.ok(stats);
})
})
.fail(function(err) {
console.log(err);
dbObject.vipUp = false;
})
I am not sure what the issue is or how to solve it.
What you are doing here is using the node native driver methods to connect and inspect the database. There is in fact "no such method" as .getUsers() here in this API or in fact in any other API.
The .getUsers() function is just a "shell helper" that is basically implemented like this:
function (args) {
var cmdObj = {usersInfo: 1};
Object.extend(cmdObj, args);
var res = this.runCommand(cmdObj);
if (!res.ok) {
var authSchemaIncompatibleCode = 69;
if (res.code == authSchemaIncompatibleCode ||
(res.code == null && res.errmsg == "no such cmd: usersInfo")) {
// Working with 2.4 schema user data
return this.system.users.find({}).toArray();
}
throw Error(res.errmsg);
}
return res.users;
}
So what you should be able to see here is that this normally wraps a "command" form, or otherwise falls back for compatibility with MongoDB 2.4 to querying the system.users collection on the current database.
Therefore, instead of calling a method that does not exist, you then need to use the .command() method instead:
mp.MongoClient.connect("mongodb://#localhost:27017/test")
.then(function(db){
db.command({ "usersInfo": 1}).then(function(users){
res.ok(users);
})
})
.fail(function(err) {
console.log(err);
})
Or in the case of connecting to a MongoDB 2.4 instance, then fetch from the .collection():
mp.MongoClient.connect("mongodb://#localhost:27017/test")
.then(function(db){
db.collection('system.users').find().toArray().then(function(users){
res.ok(users);
})
})
.fail(function(err) {
console.log(err);
})
At any rate, you really should be establishing the database connection elsewhere in your application ( or re-using the underlying driver connection from another store ), and then calling methods on the connection already establihed. This is always preferable to creating a connection on the request of the information you want to retrieve.
Also, recent versions of the node native driver support promises right out of the box. So there may be no need to configure in anything else, depending on how you intend to use it.
Is it possible for 2 Meteor.Collections to be retrieving data from 2 different mongdb database servers?
Dogs = Meteor.Collection('dogs') // mongodb://192.168.1.123:27017/dogs
Cats = Meteor.Collection('cats') // mongodb://192.168.1.124:27017/cats
Update
It is now possible to connect to remote/multiple databases:
var database = new MongoInternals.RemoteCollectionDriver("<mongo url>");
MyCollection = new Mongo.Collection("collection_name", { _driver: database });
Where <mongo_url> is a mongodb url such as mongodb://127.0.0.1:27017/meteor (with the database name)
There is one disadvantage with this at the moment: No Oplog
Old Answer
At the moment this is not possible. Each meteor app is bound to one database.
There are a few ways you can get around this but it may be more complicated that its worth:
One option - Use a separate Meteor App
In your other meteor app (example running at port 6000 on same machine). You can still have reactivity but you need to proxy inserts, removes and updates through a method call
Server:
Cats = Meteor.Collection('cats')
Meteor.publish("cats", function() {
return Cats.find();
});
Meteor.methods('updateCat, function(id, changes) {
Cats.update({_id: id}, {$set:changes});
});
Your current Meteor app:
var connection = DDP.connect("http://localhost:6000");
connection.subscribe("cats");
Cats = Meteor.Collection('cats', {connection: connection});
//To update a collection
Cats.call("updateCat", <cat_id>, <changes);
Another option - custom mongodb connection
This uses the node js mongodb native driver.
This is connecting to the database as if you would do in any other node js app.
There is no reactivity available and you can't use the new Meteor.Collection type collections.
var mongodb = Npm.require("mongodb"); //or var mongodb = Meteor.require("mongodb") //if you use npm package on atmosphere
var db = mongodb.Db;
var mongoclient = mongodb.MongoClient;
var Server = mongodb.Server;
var db_connection = new Db('cats', new Server("127.0.0.1", 27017, {auto_reconnect: false, poolSize: 4}), {w:0, native_parser: false});
db.open(function(err, db) {
//Connected to db 'cats'
db.authenticate('<db username>', '<db password>', function(err, result) {
//Can do queries here
db.close();
});
});
The answer is YES: it is possible set up multiple Meteor.Collections to be retrieving data from different mongdb database servers.
As the answer from #Akshat, you can initialize your own MongoInternals.RemoteCollectionDriver instance, through which Mongo.Collections can be created.
But here's something more to talk about. Being contrary to #Akshat answer, I find that Oplog support is still available under such circumstance.
When initializing the custom MongoInternals.RemoteCollectionDriver, DO NOT forget to specify the Oplog url:
var driver = new MongoInternals.RemoteCollectionDriver(
"mongodb://localhost:27017/db",
{
oplogUrl: "mongodb://localhost:27017/local"
});
var collection = new Mongo.Collection("Coll", {_driver: driver});
Under the hood
As described above, it is fairly simple to activate Oplog support. If you do want to know what happened beneath those two lines of code, you can continue reading the rest of the post.
In the constructor of RemoteCollectionDriver, an underlying MongoConnection will be created:
MongoInternals.RemoteCollectionDriver = function (
mongo_url, options) {
var self = this;
self.mongo = new MongoConnection(mongo_url, options);
};
The tricky part is: if MongoConnection is created with oplogUrl provided, an OplogHandle will be initialized, and starts to tail the Oplog (source code):
if (options.oplogUrl && ! Package['disable-oplog']) {
self._oplogHandle = new OplogHandle(options.oplogUrl, self.db.databaseName);
self._docFetcher = new DocFetcher(self);
}
As this blog has described: Meteor.publish internally calls Cursor.observeChanges to create an ObserveHandle instance, which automatically tracks any future changes occurred in the database.
Currently there are two kinds of observer drivers: the legacy PollingObserveDriver which takes a poll-and-diff strategy, and the OplogObseveDriver, which effectively use Oplog-tailing to monitor data changes. To decide which one to apply, observeChanges takes the following procedure (source code):
var driverClass = canUseOplog ? OplogObserveDriver : PollingObserveDriver;
observeDriver = new driverClass({
cursorDescription: cursorDescription,
mongoHandle: self,
multiplexer: multiplexer,
ordered: ordered,
matcher: matcher, // ignored by polling
sorter: sorter, // ignored by polling
_testOnlyPollCallback: callbacks._testOnlyPollCallback
});
In order to make canUseOplog true, several requirements should be met. A bare minimal one is: the underlying MongoConnection instance should have a valid OplogHandle. This is the exact reason why we need to specify oplogUrl while creating MongoConnection
This is actually possible, using an internal interface:
var d = new MongoInternals.RemoteCollectionDriver("<mongo url>");
C = new Mongo.Collection("<collection name>", { _driver: d });
I am using node.js and with the native mongodb driver (node-mongodb-native);
My current project uses node.js + now.js + mongo-db.
The system basically sends data from the browser to node.js, which is processed with haskell and later fed back to the browser again.
Via a form and node.js the text is inserted in a mongo-db collection called "messages".
A haskell thread reads the entry and stores the result in the db collection "results". This works fine.
But now I need the javascript code that waits for the result to appear in the collection results.
Pseudo code:
wait until the collection result is non-empty.
findOne() from the collection results.
delete the collection results.
I currently connect to the mongodb like this:
var mongo = require('mongodb'),
Server = mongo.Server,
Db = mongo.Db;
var server = new Server('localhost', 27017, {
auto_reconnect: true
});
var db = new Db('test', server);
My haskell knowledge is quite good but not my javascript skills.
So I did extensive searches, but I didn't get far.
glad you solved it, i was going to write something similar:
setTimeout(function(){
db.collection('results',function(coll){
coll.findOne({}, function(err, one){
if( err ) return callback(err);
coll.drop(callback); //or destroy, not really sure <-- this will drop the whole collection
});
});
} ,1000);
The solution is to use the async library.
var async = require('async');
globalCount = -1;
async.whilst(
function () {
return globalCount<1;
},
function (callback) {
console.log("inner while loop");
setTimeout(db_count(callback), 1000);
},
function (err) {
console.log(" || whilst loop finished!!");
}
);