need a little assistance with an understanding nodejs code organization,
so I'm from C++ world and suppose that didn't understand a principles.
So I need to implement some js module which should connect to MongoDB and exports a few methods for other modules: e.g. insert, update, delete.
when I write something like:
var db = MongoClient.connect(config.connectionString, {native_parser:true},function (err, db) {...});
exports.insert = function(a, b) {
// db using
//...
};
I suppose that "db" local static variable and will be initialized in any case. at the time of call "require('this module') " but seems it's not so, and db is uninitialized at the time of the call of exported functions? another question - I suppose this should be implemented using "futures" (class from c++, didn't find an analogue from js) to guaratee that db object is copmpletely constructed at the moment of the using??
So the problem I see is that you want to use DB but since DB is returned async, it may or may not be available in the exported function, hence you need to convert the connect from async to sync.
Since MongoDB driver cannot do sync, i suggest you use a wrapper, i suggest mongoskin.
https://github.com/kissjs/node-mongoskin
var mongo = require('mongoskin');
var db = mongo.db(config.connectionString, {native_parser:true});
Now this should work for you.
I had worked with C++, Java before (sometime back, not now) and now working in nodejs. I think I understood your question. Here are some key points.
Yes, Nodejs modules are somewhat like classes that they encapsulate the variables and you access only through public methods (exposed through exports). I think you are aware that there is no class implementation at all here, but it loosely maps to the behaviour.
The key difference in nodejs is the asynchronous nature of resource instantiation. By this, I mean if there are 2 statements stmt1 and stmt2, if stmt1 is called and takes time, nodejs does not wait for it to end (that is synchronous behaviour), instead it moves on to stmt2. In pre-nodejs world, we assume that reaching stmt2 means stmt1 is complete.
So, what is the workaround? How to ensure you do something after db connection is obtained. If your code is not making db calls immediately, you could assume that connection will be through. Or if you immediately want to invoke db, you write the code on a callback. Mongo exposes events called 'open' and 'error'. You can use this to ensure connection is open. Also it is best practise to track error event.
db.on('error', console.error.bind(console, 'connection error'));
db.once('open', function callback() {
console.log("Connection with database succeeded.");
// put your code
});
I am not aware of C++ future and so cannot comment on that.
Hope this helps !
[Updated] To add example
You could have db.js for setting up db connection and expose Mongoose object to create models.
'use strict';
var Mongoose = require('mongoose'),
Config = require('./config');
Mongoose.connect(Config.database.url);
var db = Mongoose.connection;
db.on('error', console.error.bind(console, 'connection error'));
db.once('open', function callback() {
console.log("Connection with database succeeded.");
});
exports.Mongoose = Mongoose;
exports.db = db;
You can include db.js in the server.js like
var DB = require('db.js');
which will do the initialisation.
You then use mongoose (mongoose is a Object relational mapper to work with mongo and highly recommended) to get models of database objects as shown below.
//userModel.js
var mongoose = require('mongoose'),
Schema = mongoose.Schema,
var UserSchema = new Schema({
uid : {type : Number, required: false}
, email : {type : String, get: toLower, set: toLower, required: true, index: { unique: true } }
, passwd : {type : String, required: false}
);
var user = mongoose.model('user', UserSchema);
module.exports = {
User : user
};
For more information on mongoose, you can refer http://mongoosejs.com
The db is generally not closed as I use in web environment and is always on. There is db connection pooling maintained and connections are reused optimally. I saw noticed a thread in SO which adds more details. Why is it recommended not to close a MongoDB connection anywhere in Node.js code?
Related
I've been working on a server and a push notification daemon that will both run simultaneously and interact with the same database. The idea behind this is that if one goes down, the other will still function.
I normally use Swift but for this project I'm writing it in Node, using Mongoose as my database. I've created a helper class that I import in both my server.js file and my notifier.js file.
const Mongoose = require('mongoose');
const Device = require('./device'); // This is a Schema
var uri = 'mongodb://localhost/devices';
function Database() {
Mongoose.connect(uri, { useMongoClient: true }, function(err) {
console.log('connected: ' + err);
});
}
Database.prototype.findDevice = function(params, callback) {
Device.findOne(params, function(err, device) {
// etc...
});
};
module.exports = Database;
Then separately from both server.js and notifier.js I create objects and query the database:
const Database = require('./db');
const db = new Database();
db.findDevice(params, function(err, device) {
// Simplified, but I edit and save things back to the database via db
device.token = 'blah';
device.save();
});
Is this safe to do? When working with Swift (and Objective-C) I'm always concerned about making things thread safe. Is this a concern? Should I be worried about race conditions and modifying the same files at the same time?
Also, bonus question: How does Mongoose share a connection between files (or processes?). For example Mongoose.connection.readyState returns the same thing from different files.
The short answer is "safe enough."
The long answer has to do with understanding what sort of consistency guarantees your system needs, how you've configured MongoDB, and whether there's any sharding or replication going on.
For the latter, you'll want to read about atomicity and consistency and perhaps also peek at write concern.
A good way to answer these questions, even when you think you've figured it out, is to test scenarios: Hammer a duplicate of your system with fake data and events and see if what happen is OK or not.
As per example (db.js)
const pg = require('pg');
const client_config = {...};
const pool = new pg.Pool(client_config);
pool.on('error', function(err, client) {
console.error('idle client error', err.mesae, err.stack);
});
module.exports.query = function(text, values, callback) {
return pool.query(text, values, callback);
};
module.exports.connect = function(callback) {
return pool.connect(callback);
};
and within an express (generated) application, do I have to initiate/require the Pool (db.js) in my app.js/on app start-up or do I simply require the db.js within my data models (respectively required in my routes)? Intuitively I would initiate the Pool on start-up rather than on each connection to the routes to avoid multiple initiations, but I am fairly new to Node.js.
Scroll a little further -- there are usage examples.
The reason this works is thanks to Node's module caching. The first time db.js is required, all the init code executes immediately. Subsequent require calls return the already-initialized module from the cache so the pool is already connected. In Express, you can avoid requiring db.js all over the place by using app.set('db', db); to attach the module to the Express application. You can then invoke req.app.get('db').query(...) in your route code.
If your data needs are complex enough to involve models, you may want to look into higher-level data access libraries since pg is more of a driver (think JDBC if you've done any Java). There are a lot of options ranging from minimal data mappers (I maintain MassiveJS) to query builders (Knex) to full-scale ORMs (Sequelize).
I'm building a larger web app, where the routes are divided into separate files.
All routes need a connection to the db, and therefore all of them require mongoskin, which is the module I'm using for MongoDb. Like this:
var mongo = require('mongoskin');
But soon after I realised that only require the mongoskin wasn't enough for the routes to be able to talk to the db. Because in my main app.js file I also made additional "configurations".
db = mongo.db('mongodb://localhost/dbName', {native_parser:true});
db.open(function(err) {
if (!err) {
console.log('Connected to mongodb://localhost/dbName');
}
});
db.bind('clients');
db.bind('invoices');
I needed this db object to be shared aswell...
My first attempt was to wrap the route file in a exported function that takes an argument. This argument is passed in when I require the routes.js in my main app.js. This worked out fine, but I wasn't really fond of this solution... I think it became a bit messy.
My second approach, which I'm using right now, is to make a separate module of the whole db object.
var mongo = require('mongoskin');
var db = null;
module.exports = {
initAndGetDb: function () {
db = mongo.db('mongodb://localhost/dbName', {native_parser:true});
db.open(function(err) {
if (!err) {
console.log('Connected to mongodb://localhost/dbName');
}
});
db.bind('clients');
db.bind('invoices');
return(db);
},
getDb: function () {
return(db);
}
};
In my main app.js
var db = require('./db').initAndGetDb();
And in my routes.js
var db = require('../db').getDb();
Question: Is this approach a good working solution for sharing a db connection (and maybe other things in a similar fashion)? If you can see any problem with this, please let me know...
Overall I think this is fine, but you could simplify it to just:
//your db.js module
var mongo = require('mongoskin');
var db = mongo.db('mongodb://localhost/dbName', {native_parser:true});
db.bind('clients');
db.bind('invoices');
db.open(function(err) {
if (err) {
console.error('Could not connect to db', err);
return;
}
console.log('Connected to mongodb://localhost/dbName');
});
module.exports = db;
The first time your code does require("./db");, the top-level code in db.js will run and connect to the db. When other modules require it, they will get access to the db without re-running the top level code and reconnecting.
Note that to be truly ready for production you would need to enhance this with:
get DB connection details from some configuration system (env vars or a helper module)
More robust logging
Graceful handling of disconnects and reconnects while the app is running
Graceful handling of the db being down when the web app starts
retry/backoff logic around connecting/reconnecting
Decide what the webapp does when it can't reach the DB. Show a fail whale page or exit the process.
Is it possible for 2 Meteor.Collections to be retrieving data from 2 different mongdb database servers?
Dogs = Meteor.Collection('dogs') // mongodb://192.168.1.123:27017/dogs
Cats = Meteor.Collection('cats') // mongodb://192.168.1.124:27017/cats
Update
It is now possible to connect to remote/multiple databases:
var database = new MongoInternals.RemoteCollectionDriver("<mongo url>");
MyCollection = new Mongo.Collection("collection_name", { _driver: database });
Where <mongo_url> is a mongodb url such as mongodb://127.0.0.1:27017/meteor (with the database name)
There is one disadvantage with this at the moment: No Oplog
Old Answer
At the moment this is not possible. Each meteor app is bound to one database.
There are a few ways you can get around this but it may be more complicated that its worth:
One option - Use a separate Meteor App
In your other meteor app (example running at port 6000 on same machine). You can still have reactivity but you need to proxy inserts, removes and updates through a method call
Server:
Cats = Meteor.Collection('cats')
Meteor.publish("cats", function() {
return Cats.find();
});
Meteor.methods('updateCat, function(id, changes) {
Cats.update({_id: id}, {$set:changes});
});
Your current Meteor app:
var connection = DDP.connect("http://localhost:6000");
connection.subscribe("cats");
Cats = Meteor.Collection('cats', {connection: connection});
//To update a collection
Cats.call("updateCat", <cat_id>, <changes);
Another option - custom mongodb connection
This uses the node js mongodb native driver.
This is connecting to the database as if you would do in any other node js app.
There is no reactivity available and you can't use the new Meteor.Collection type collections.
var mongodb = Npm.require("mongodb"); //or var mongodb = Meteor.require("mongodb") //if you use npm package on atmosphere
var db = mongodb.Db;
var mongoclient = mongodb.MongoClient;
var Server = mongodb.Server;
var db_connection = new Db('cats', new Server("127.0.0.1", 27017, {auto_reconnect: false, poolSize: 4}), {w:0, native_parser: false});
db.open(function(err, db) {
//Connected to db 'cats'
db.authenticate('<db username>', '<db password>', function(err, result) {
//Can do queries here
db.close();
});
});
The answer is YES: it is possible set up multiple Meteor.Collections to be retrieving data from different mongdb database servers.
As the answer from #Akshat, you can initialize your own MongoInternals.RemoteCollectionDriver instance, through which Mongo.Collections can be created.
But here's something more to talk about. Being contrary to #Akshat answer, I find that Oplog support is still available under such circumstance.
When initializing the custom MongoInternals.RemoteCollectionDriver, DO NOT forget to specify the Oplog url:
var driver = new MongoInternals.RemoteCollectionDriver(
"mongodb://localhost:27017/db",
{
oplogUrl: "mongodb://localhost:27017/local"
});
var collection = new Mongo.Collection("Coll", {_driver: driver});
Under the hood
As described above, it is fairly simple to activate Oplog support. If you do want to know what happened beneath those two lines of code, you can continue reading the rest of the post.
In the constructor of RemoteCollectionDriver, an underlying MongoConnection will be created:
MongoInternals.RemoteCollectionDriver = function (
mongo_url, options) {
var self = this;
self.mongo = new MongoConnection(mongo_url, options);
};
The tricky part is: if MongoConnection is created with oplogUrl provided, an OplogHandle will be initialized, and starts to tail the Oplog (source code):
if (options.oplogUrl && ! Package['disable-oplog']) {
self._oplogHandle = new OplogHandle(options.oplogUrl, self.db.databaseName);
self._docFetcher = new DocFetcher(self);
}
As this blog has described: Meteor.publish internally calls Cursor.observeChanges to create an ObserveHandle instance, which automatically tracks any future changes occurred in the database.
Currently there are two kinds of observer drivers: the legacy PollingObserveDriver which takes a poll-and-diff strategy, and the OplogObseveDriver, which effectively use Oplog-tailing to monitor data changes. To decide which one to apply, observeChanges takes the following procedure (source code):
var driverClass = canUseOplog ? OplogObserveDriver : PollingObserveDriver;
observeDriver = new driverClass({
cursorDescription: cursorDescription,
mongoHandle: self,
multiplexer: multiplexer,
ordered: ordered,
matcher: matcher, // ignored by polling
sorter: sorter, // ignored by polling
_testOnlyPollCallback: callbacks._testOnlyPollCallback
});
In order to make canUseOplog true, several requirements should be met. A bare minimal one is: the underlying MongoConnection instance should have a valid OplogHandle. This is the exact reason why we need to specify oplogUrl while creating MongoConnection
This is actually possible, using an internal interface:
var d = new MongoInternals.RemoteCollectionDriver("<mongo url>");
C = new Mongo.Collection("<collection name>", { _driver: d });
In the following code, are the "db" variables in session.js and user.js referencing to the same object in db.js, or are they copies of it (making separate connections to my db server)?
// db.js
var mongojs = require('mongojs');
var db = mongojs('test', ['users', 'sessions']);
module.exports.database = db;
// session.js
var db = require('../db.js').database;
......
// user.js
var db = require('../db.js').database;
......
Thanks!
A required module is cached, so they will both point to the same object. Relevant Node.js documentation: Modules caching docs.
Every call to require('../db.js') returns the same object, so in your case there would just be a single database connection pool created.
Note that database is actually a pool of connections (5 by default) that can be freely shared across your code so this is likely what you want.
See the docs here.