I am currently working on building a backend using Express and MongoDB Native Node. I have been researching attempting to find the best "best practice" for managing connections to a Mongo database and using that connection across the app.
My current solution is working and I get desired results via tests in Postman. I have come up with this after being unable to find concrete answers on handling the connections with MongoDB 3.x (without Mongoose) and still modular.
Would someone be willing to give some feedback on my current solution?
My main concern would be that this setup would not be performant. I suspect it may not be, potentially due to opening/closing connections frequently, but I am not sure if the way I am doing it is good or bad practice.
I created a db.js file to serve my connection:
const assert = require("assert");
const MongoClient = require("mongodb").MongoClient;
const base = process.env.PWD;
const config = require(base + "/config");
let db;
let client;
const connect = async () => {
const url = config.url
const dbName = config.dbName
client = await MongoClient.connect(url)
db = await client.db(dbName)
return db
}
const disconnect = () => {
client.close()
}
module.exports = {
connect: connect,
disconnect: disconnect
}
I then set the routes for my 'todos' in index.js inside of my todos folder. Following a best practice suggestion to have all component files in their own folders(open to feedback on folder structure):
const express = require('express'),
base = process.env.PWD,
router = express.Router(),
todos = require(base + '/todos/todosController')
/* GET All Todos */
router.get('/all', todos.getTodos)
/* GET One Todo */
router.get('/todo/:id', todos.getTodo)
/* POST One Todo */
router.post('/todo/:id', todos.addTodo)
/* DELETE One Todo */
router.delete('/todo/:id', todos.deleteTodo)
module.exports = router;
Lastly the actual todosController.js which requires db.js
This is where I suspect some improvement could happen but I am just not sure. I connect within the routes via async function and await the connection and assign it to db I do my CRUD queries (all currently working properly) and then disconnect at the end.
If this is considered performant and a good practice I am happy with that answer but if there is a way to do this better with current driver and syntax I would be happy for any feedback.
'use strict';
const base = process.env.PWD,
client = require(base + '/db.js'),
assert = require('assert')
let db
const getTodos = async (req, res) => {
db = await client.connect()
const collection = await db.collection('documents')
// Find all todos
collection.find({}).toArray((err, todos) => {
assert.equal(err, null)
res.status(200).json(todos)
})
client.disconnect()
}
It seems that this is a common misconception that opening and closing a connection on every request is more efficient. Opening connection is expensive and this is one of the reasons for the existence of the connection pools. MongoDB supports those and you should consider them.
Here is an article on the subject of Express/MongoDB connection handling which starts right away with:
A common mistake developers make when connecting to the database is to
call MongoClient.connect() in every route handler to get a database
connection.
Related
I have an express route that needs to retrieve a row from a db, check if a field has a value set and if not set it. The row data then gets sent back to the client.
I know that node runs on a single thread but I/O operations do actually run asynchronously so I think I may have a problem if whilst the first client is waiting to write to db, a second client comes and reads a null value and performs the write a second time.
I can't have this happen as the value written is a shared value that there can only be one of.
Am I correct that this could happen and if so what is a recommended way to handle this?
Thanks.
let express = require('express');
let router = express.Router();
router.post('/getRoomStateByRoomUrl', async (req, res, next) => {
const roomUrl = req.body.room_url;
try {
//READ FROM DB
const roomState = await RoomStateModel.getRoomStateByRoomUrl(roomUrl);
if(!roomState.tokboxSessionID) {
const newSessionID = await TokboxService.createSession();
//WRITE TO DB
await RoomStateModel.setTokboxSessionID(newSessionID);
roomState.tokboxSessionID = newSessionID;
}
res.status(200).json(roomState);
} catch (error) {
next(error);
}
});
I am using mlab free tier, and I'm trying to use change streams in my app. Here's my code:
const config = require('../globals/config');
const MongoClient = require('mongodb').MongoClient;
MongoClient.connect(config.db, {useNewUrlParser: true}, function (err, client) {
const db = client.db('web-project-studio-db');
// Connect using MongoClient
var filter = [{$match: {'_id': user._id}}];
db.collection('accounts').watch(filter).on('change', data => {
console.log(data);
});
});
This says:
MongoError: Majority read concern requested, but it is not supported by the storage engine.
Is there something I should enable in mlab settings, or did I just mess something up?
To use watch you need to use replica set which is not part of mLab free plan.
I've been working on a server and a push notification daemon that will both run simultaneously and interact with the same database. The idea behind this is that if one goes down, the other will still function.
I normally use Swift but for this project I'm writing it in Node, using Mongoose as my database. I've created a helper class that I import in both my server.js file and my notifier.js file.
const Mongoose = require('mongoose');
const Device = require('./device'); // This is a Schema
var uri = 'mongodb://localhost/devices';
function Database() {
Mongoose.connect(uri, { useMongoClient: true }, function(err) {
console.log('connected: ' + err);
});
}
Database.prototype.findDevice = function(params, callback) {
Device.findOne(params, function(err, device) {
// etc...
});
};
module.exports = Database;
Then separately from both server.js and notifier.js I create objects and query the database:
const Database = require('./db');
const db = new Database();
db.findDevice(params, function(err, device) {
// Simplified, but I edit and save things back to the database via db
device.token = 'blah';
device.save();
});
Is this safe to do? When working with Swift (and Objective-C) I'm always concerned about making things thread safe. Is this a concern? Should I be worried about race conditions and modifying the same files at the same time?
Also, bonus question: How does Mongoose share a connection between files (or processes?). For example Mongoose.connection.readyState returns the same thing from different files.
The short answer is "safe enough."
The long answer has to do with understanding what sort of consistency guarantees your system needs, how you've configured MongoDB, and whether there's any sharding or replication going on.
For the latter, you'll want to read about atomicity and consistency and perhaps also peek at write concern.
A good way to answer these questions, even when you think you've figured it out, is to test scenarios: Hammer a duplicate of your system with fake data and events and see if what happen is OK or not.
As per example (db.js)
const pg = require('pg');
const client_config = {...};
const pool = new pg.Pool(client_config);
pool.on('error', function(err, client) {
console.error('idle client error', err.mesae, err.stack);
});
module.exports.query = function(text, values, callback) {
return pool.query(text, values, callback);
};
module.exports.connect = function(callback) {
return pool.connect(callback);
};
and within an express (generated) application, do I have to initiate/require the Pool (db.js) in my app.js/on app start-up or do I simply require the db.js within my data models (respectively required in my routes)? Intuitively I would initiate the Pool on start-up rather than on each connection to the routes to avoid multiple initiations, but I am fairly new to Node.js.
Scroll a little further -- there are usage examples.
The reason this works is thanks to Node's module caching. The first time db.js is required, all the init code executes immediately. Subsequent require calls return the already-initialized module from the cache so the pool is already connected. In Express, you can avoid requiring db.js all over the place by using app.set('db', db); to attach the module to the Express application. You can then invoke req.app.get('db').query(...) in your route code.
If your data needs are complex enough to involve models, you may want to look into higher-level data access libraries since pg is more of a driver (think JDBC if you've done any Java). There are a lot of options ranging from minimal data mappers (I maintain MassiveJS) to query builders (Knex) to full-scale ORMs (Sequelize).
I'm building a larger web app, where the routes are divided into separate files.
All routes need a connection to the db, and therefore all of them require mongoskin, which is the module I'm using for MongoDb. Like this:
var mongo = require('mongoskin');
But soon after I realised that only require the mongoskin wasn't enough for the routes to be able to talk to the db. Because in my main app.js file I also made additional "configurations".
db = mongo.db('mongodb://localhost/dbName', {native_parser:true});
db.open(function(err) {
if (!err) {
console.log('Connected to mongodb://localhost/dbName');
}
});
db.bind('clients');
db.bind('invoices');
I needed this db object to be shared aswell...
My first attempt was to wrap the route file in a exported function that takes an argument. This argument is passed in when I require the routes.js in my main app.js. This worked out fine, but I wasn't really fond of this solution... I think it became a bit messy.
My second approach, which I'm using right now, is to make a separate module of the whole db object.
var mongo = require('mongoskin');
var db = null;
module.exports = {
initAndGetDb: function () {
db = mongo.db('mongodb://localhost/dbName', {native_parser:true});
db.open(function(err) {
if (!err) {
console.log('Connected to mongodb://localhost/dbName');
}
});
db.bind('clients');
db.bind('invoices');
return(db);
},
getDb: function () {
return(db);
}
};
In my main app.js
var db = require('./db').initAndGetDb();
And in my routes.js
var db = require('../db').getDb();
Question: Is this approach a good working solution for sharing a db connection (and maybe other things in a similar fashion)? If you can see any problem with this, please let me know...
Overall I think this is fine, but you could simplify it to just:
//your db.js module
var mongo = require('mongoskin');
var db = mongo.db('mongodb://localhost/dbName', {native_parser:true});
db.bind('clients');
db.bind('invoices');
db.open(function(err) {
if (err) {
console.error('Could not connect to db', err);
return;
}
console.log('Connected to mongodb://localhost/dbName');
});
module.exports = db;
The first time your code does require("./db");, the top-level code in db.js will run and connect to the db. When other modules require it, they will get access to the db without re-running the top level code and reconnecting.
Note that to be truly ready for production you would need to enhance this with:
get DB connection details from some configuration system (env vars or a helper module)
More robust logging
Graceful handling of disconnects and reconnects while the app is running
Graceful handling of the db being down when the web app starts
retry/backoff logic around connecting/reconnecting
Decide what the webapp does when it can't reach the DB. Show a fail whale page or exit the process.