I'm doing a Node.js project that contains sub projects. One sub project will have one Mongodb database and Mongoose will be use for wrapping and querying db. But the problem is
Mongoose doesn't allow to use multiple databases in single mongoose instance as the models are build on one connection.
To use multiple mongoose instances, Node.js doesn't allow multiple module instances as it has caching system in require(). I know disable module caching in Node.js but I think it is not the good solution as it is only need for mongoose.
I've tried to use createConnection() and openSet() in mongoose, but it was not the solution.
I've tried to deep copy the mongoose instance (http://blog.imaginea.com/deep-copy-in-javascript/) to pass new mongoose instances to the sub project, but it throwing RangeError: Maximum call stack size exceeded.
I want to know is there anyways to use multiple database with mongoose or any workaround for this problem? Because I think mongoose is quite easy and fast. Or any other modules as recommendations?
According to the fine manual, createConnection() can be used to connect to multiple databases.
However, you need to create separate models for each connection/database:
var conn = mongoose.createConnection('mongodb://localhost/testA');
var conn2 = mongoose.createConnection('mongodb://localhost/testB');
// stored in 'testA' database
var ModelA = conn.model('Model', new mongoose.Schema({
title : { type : String, default : 'model in testA database' }
}));
// stored in 'testB' database
var ModelB = conn2.model('Model', new mongoose.Schema({
title : { type : String, default : 'model in testB database' }
}));
I'm pretty sure that you can share the schema between them, but you have to check to make sure.
Pretty late but this might help someone. The current answers assumes you are using the same file for your connections and models.
In real life, there is a high chance that you are splitting your models into different files. You can use something like this in your main file:
mongoose.connect('mongodb://localhost/default');
const db = mongoose.connection;
db.on('error', console.error.bind(console, 'connection error:'));
db.once('open', () => {
console.log('connected');
});
which is just how it is described in the docs. And then in your model files, do something like the following:
import mongoose, { Schema } from 'mongoose';
const userInfoSchema = new Schema({
createdAt: {
type: Date,
required: true,
default: new Date(),
},
// ...other fields
});
const myDB = mongoose.connection.useDb('myDB');
const UserInfo = myDB.model('userInfo', userInfoSchema);
export default UserInfo;
Where myDB is your database name.
One thing you can do is, you might have subfolders for each projects. So, install mongoose in that subfolders and require() mongoose from own folders in each sub applications. Not from the project root or from global. So one sub project, one mongoose installation and one mongoose instance.
-app_root/
--foo_app/
---db_access.js
---foo_db_connect.js
---node_modules/
----mongoose/
--bar_app/
---db_access.js
---bar_db_connect.js
---node_modules/
----mongoose/
In foo_db_connect.js
var mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/foo_db');
module.exports = exports = mongoose;
In bar_db_connect.js
var mongoose = require('mongoose');
mongoose.connect('mongodb://localhost/bar_db');
module.exports = exports = mongoose;
In db_access.js files
var mongoose = require("./foo_db_connect.js"); // bar_db_connect.js for bar app
Now, you can access multiple databases with mongoose.
As an alternative approach, Mongoose does export a constructor for a new instance on the default instance. So something like this is possible.
var Mongoose = require('mongoose').Mongoose;
var instance1 = new Mongoose();
instance1.connect('foo');
var instance2 = new Mongoose();
instance2.connect('bar');
This is very useful when working with separate data sources, and also when you want to have a separate database context for each user or request. You will need to be careful, as it is possible to create a LOT of connections when doing this. Make sure to call disconnect() when instances are not needed, and also to limit the pool size created by each instance.
Mongoose and multiple database in single node.js project
use useDb to solve this issue
example
//product databse
const myDB = mongoose.connection.useDb('product');
module.exports = myDB.model("Snack", snackSchema);
//user databse
const myDB = mongoose.connection.useDb('user');
module.exports = myDB.model("User", userSchema);
A bit optimized(for me atleast) solution. write this to a file db.js and require this to wherever required and call it with a function call and you are good to go.
const MongoClient = require('mongodb').MongoClient;
async function getConnections(url,db){
return new Promise((resolve,reject)=>{
MongoClient.connect(url, { useUnifiedTopology: true },function(err, client) {
if(err) { console.error(err)
resolve(false);
}
else{
resolve(client.db(db));
}
})
});
}
module.exports = async function(){
let dbs = [];
dbs['db1'] = await getConnections('mongodb://localhost:27017/','db1');
dbs['db2'] = await getConnections('mongodb://localhost:27017/','db2');
return dbs;
};
I have been using this method and it works great for me until now.
const mongoose = require('mongoose');
function makeNewConnection(uri) {
const db = mongoose.createConnection(uri, {
useNewUrlParser: true,
useUnifiedTopology: true
});
db.on('error', function (error) {
console.log(`MongoDB :: connection ${this.name} ${JSON.stringify(error)}`);
db.close().catch(() => console.log(`MongoDB :: failed to close connection ${this.name}`));
});
db.on('connected', function () {
mongoose.set('debug', function (col, method, query, doc) {
console.log(`MongoDB :: ${this.conn.name} ${col}.${method}(${JSON.stringify(query)},${JSON.stringify(doc)})`);
});
console.log(`MongoDB :: connected ${this.name}`);
});
db.on('disconnected', function () {
console.log(`MongoDB :: disconnected ${this.name}`);
});
return db;
}
// Use
const db1 = makeNewConnection(MONGO_URI_DB1);
const db2 = makeNewConnection(MONGO_URI_DB2);
module.exports = {
db1,
db2
}
Related
I have a problem with the approach I use to connect to Mondo DB.
I use the following method:
import { Db, MongoClient } from "mongodb";
let cachedConnection: { client: MongoClient; db: Db } | null = null;
export async function connectToDatabase(mongoUri?: string, database?: string) {
if (!mongoUri) {
throw new Error(
"Please define the MONGO_URI environment variable inside .env.local"
);
}
if (!database) {
throw new Error(
"Please define the DATABASE environment variable inside .env.local"
);
}
if (cachedConnection) return cachedConnection;
cachedConnection = await MongoClient.connect(mongoUri, {
useNewUrlParser: true,
useUnifiedTopology: true,
}).then((client) => ({
client,
db: client.db(database),
}));
return cachedConnection!;
}
Everytime I need to connect to MongoDB I do as follows:
const { db } = await connectToDatabase(config.URI, config.USERS_DATABASE);
const myUniversity = await db
.collection(config.MY_COLLECTION)
.findOne({})
Everything seems ok, so what is the problem?
The problem is that the connections to my DB don't close after I use them. In fact I thought that my server is stateless so after every time i use my DB, the connections end. But it is not true! They stay alive, and after few hours of using my app mongo atlas sends me an email saying that the limit is exceeded.
As you can see in this screenshot, this chart is ever growing. That means that connections stay on and they accumulate. How do you think I can solve this problem?
Keep in mind that it uses cachedConnection only if I use the same connection. If I call a different API from the first one it creates another connection and it doesn't enter in if (cachedConnection) block, but it goes forward till the end.
You can try this simple demo which will allow you to use the same connection throughout the application in different modules. There are three modules: the index.js is the starter program, the dbaccess.js is where you have code to create and maintain a connection which can be used again and again, and a apis.js module where you use the database connection to retrieve data.
index.js:
const express = require('express');
const mongo = require('./dbaccess');
const apis = require('./apis');
const app = express();
const init = async () => {
await mongo.connect();
app.listen(3000);
apis(app, mongo);
};
init();
dbaccess.js:
const { MongoClient } = require('mongodb');
class Mongo {
constructor() {
this.client = new MongoClient("mongodb://127.0.0.1:27017/", {
useNewUrlParser: true,
useUnifiedTopology: true
});
}
async connect() {
await this.client.connect();
console.log('Connected to MongoDB server.');
this.db = this.client.db('test');
console.log('Database:', this.db.databaseName);
}
}
module.exports = new Mongo();
apis.js:
module.exports = function(app, mongo) {
app.get('/', function(req, res) {
mongo.db.collection('users').find().limit(1).toArray(function(err, result) {
res.send('Doc: ' + JSON.stringify(result));
});
});
}
Change the appropriate values in the url, database name and collection name before trying.
So, I have read other topics here in StackOverflow that try to touch on this but don't have a clear solution to the problem.
First I created a config file for the mongo client which is exported from it.
const { MongoClient } = require('mongodb');
const authMechanism = 'SCRAM-SHA-1';
const user = encodeURIComponent(process.env.MONGODB_USER);
const password = encodeURIComponent(process.env.MONGODB_PASSWORD);
const uri = `mongodb://${user}:${password}#${process.env.MONGODB_URI}?authMechanism=${authMechanism}`;
const client = new MongoClient(uri, {
useUnifiedTopology: true,
useNewUrlParser: true,
loggerLevel: 'info',
});
module.exports = client;
From there I understand that you must have mongo client initialised once, and only once, before you listen to your application, hence I have created this index.js (entry point to the app) file that does that requiring the typical app.js where all the node config is.
const app = require('./app');
const db = require('../configs/db/db-config');
const port = process.env.PORT;
db.connect((err, client) => {
if (err) {
throw err;
}
const database = client.db('dbnamegoeshere');
app.listen(port, () => console.log(`Listening on port ${port}...`));
});
Now, in order for me to reuse that db anywhere I want to make queries or whatever, what is the best practice? How could I add it globally? would adding it globally affect the performance or be a bad practice?
I have seen other examples where people perform these two tasks using a class but yet again all in the same file, not with an export or a global.
One final question, where, and why should I close the db client connection.
Thank you.
I think adding the db connection object to the global object works perfectly. And if you are worried about performance, just reassign global vars to local ones.
//dbconnection.js
const debug = require('debug')('someapp:mongo');
const mongoClient = require('mongodb').MongoClient;
const mongoOptions = {};
const mongoUrl = process.env.MONGO_URL || "mongodb://localhost:27017/dbname";
function callback(err, r){
debug("callback: ", err, r);
};
module.exports = function () {
mongoClient.connect(mongoUrl, mongoOptions, (err, client) => {
if(err){
debug("MongoDB connection error: ", err);
throw err;
};
const db = global.db = client.db();
});
};
And just use it like so in your root application file, once added to the file the db connection should be globally available in other files in your project.
//server.js
require("./dbconnection")();
const userDB = global.db.collection("Users");
userDB.find({}).toArray((err, items)=>{});
To make my code more readable, I'm trying to move all database related code into a single file. and use Sequelize as ORM. I would like that this file, when included provide a ready to use Database. Tables schemas are also managed by Sequelize which is why I use the sync() method to create the tables on the first run. Unfortunately, when I run the application for the first time, I get an error that the table doesn't exist when using this code:
File: test.js
const database = require('./dbInit');
(async () => {
await database.testTable.max('id').then((maxId) => {
console.log(maxId);
});
})();
File: dbInit.js
const Sequelize = require('sequelize');
const sequelize = new Sequelize('mysql://root:root#localhost:3306/test');
const testTable = sequelize.import('testTable');
const database = {
sequelize: sequelize,
testTable: testTable,
};
sequelize
.authenticate()
.then(() => {
console.log('Connection to the database has been established successfully.');
})
.catch(error => {
console.error(error);
});
sequelize.sync();
module.exports = database;
File: testTable.js
const Sequelize = require('sequelize');
module.exports = (sequelize, DataTypes) => {
return sequelize.define('testTable',
{
id: {
type: Sequelize.BIGINT(19).UNSIGNED,
primaryKey: true,
autoIncrement: false,
}
}
);
};
When I run the code as is, without tables created, I can see from the logs that the query is run before the connection to the database is available:
> node .\test.js
Executing (default): SELECT 1+1 AS result
Executing (default): SELECT max(`id`) AS `max` FROM `testTables` AS `testTable`;
Connection to the database has been established successfully.
(node:1572) UnhandledPromiseRejectionWarning: SequelizeDatabaseError: Table 'test.testtables' doesn't exist
I have found a way to make it work by adding this like, just before the call to the DB (in test.js before the max('id') call):
await database.sequelize.sync();
Is there any other way to have the dbInit module completely independent and not having to add this sync() call inside all other files which will require database connectivity?
I've looked for sync module loading but it doesn't seem an option yet.
Because of async behavior all of ops that You want to do:
Connect
Sync
Do DB operations
You've to make it following way:
put model files to: db folder as: db/schemas/User.js
and make module file for db: db/index.js
const Sequelize = require('sequelize');
const sequelize = new Sequelize('mysql://root:root#localhost:3306/test');
const connect = async () => {
try {
await sequelize.authenticate();
await sequelize.sync();
console.log('Connection to the database has been established successfully.');
}
catch (error) {
console.error(error.message);
process.exit(-1);
}
});
const model = name => database.models[name];
const User = sequelize.import('./schemas/User');
const database = {
sequelize: sequelize,
models: {User},
connect,
model
};
module.exports = database;
and in test.js:
const db = require('./db');
(async () => {
await db.connect();
const User = db.model('User');
const id = await User.max('id');
console.log(id);
})();
P.S. forget about examples that used in web apps when developer does not care when db will connect and when express app will listen on port.
Your question is different - You want to do db query immediately, so You've to make sure connection and sync established successfully.
You can follow up my github repo Sequelize-DemoApp. It's a fully working full stack application made especially to demonstrate and understand Sequelize.js and it's integration with nodejs
I'm new to MongoDB.
When I create my node.js server I use only one DB connection (on start I connect to it).
But imagine: I have one database with some generic tables, and more databases - each for a custom client.
How can I create those DB at runtime?
start.js:
const mongoose = require("mongoose");
// import environmental variables from variables.env file
require("dotenv").config({ path: "variables.env" });
mongoose.connect(process.env.DATABASE);
mongoose.Promise = global.Promise;
mongoose.connection.on("error", err => {
console.error(`🚫 → ${err.message}`);
});
require("./models/MaintenanceType");
require("./models/Maintenance");
const app = require("./app");
app.set("port", process.env.PORT || 7777);
const server = app.listen(app.get("port"), () => {
console.log('started');
});
variables.env (example):
NODE_ENV=development
DATABASE=mongodb://db:qwe123#sometest.server.com:412345/webtest
PORT=1234
SECRET=webtest
KEY=webtestcom
and controller:
const mongoose = require("mongoose");
const Maintenance = mongoose.model("Maintenance");
exports.createMaintenance = async (req, res) => {
const maintenance = await new Maintenance(req.body).save();
// ALSO create a db and table if not exists for this client and use it somehow
res.json(maintenance);
};
is it possible to do?
You can create new connection
mongoose.connect('URI_FOR_ANOTHER_DATABASE')
But it's bad idea to create new connections, so the driver has a feature to use existing connections to query another database, for this purpose you can check useDb() method as shown here
I'm building an app using node-webkit, based on expressjs and mongoose. I'm new to basically all of this.
I've got a mongoDb hosted online and i'm try to use it in my app, but i'm missing something
I created in model folder db.js, where i connect with the db
var mongoose = require('mongoose');
mongoose.connect('mongodb://user:password#ds012345.mlab.com:port/mydb') //this isn't the real link
then my model, clients.js
var mongoose = require('mongoose');
var Schema = mongoose.Schema;
var clientSchema = new Schema ({
name: String,
//other fields
});
var client = mongoose.model('client', clientSchema);
module.exports = client;
Then, in my app.js
var db = require('./model/db')
I'm also using routes, so in my index.js i got
var client = require('../model/clients')
But i cannot use any function (save, find, ecc.), i can just create models.
I think I'm not connecting in the right way all the modules, i was previously using diskdb and i connected to it in my index.js, but i tried in the same way and it doesn't work anyway.
Also, when i build the app, my mongoose connection status is 2.
Here are a few things:
what is ecc? you should connect to something like this: mongoose.connect('mongodb://localhost:27017/test');
27017 is the default port for MongoDB and test is the name of your database. Also make sure you start mongo server with mongod then run mongo console mongo.
Your field should specify type of the data:
var clientSchema = new Schema ({
name: String,
age: Number
});
So you want to save the document into database:
var client = mongoose.model('client', clientSchema);
var data = {
nome: 'something'
};
var user = new client(data);
user.save(function(err) {
if(err) console.log(err);
});
In your route, you can do something like this to query back and send data back to the req:
var express = require('express');
var router = express.Router();
var clientSchema = require('../models/clientSchema');
router.get('/', function(req, res, next) {
UserSchema.find({} , function(err, data) {
if (err) console.log(err);
res.render('index', {
data: data
});
});
});
module.exports = router;
Hope this help!