I have a problem with the approach I use to connect to Mondo DB.
I use the following method:
import { Db, MongoClient } from "mongodb";
let cachedConnection: { client: MongoClient; db: Db } | null = null;
export async function connectToDatabase(mongoUri?: string, database?: string) {
if (!mongoUri) {
throw new Error(
"Please define the MONGO_URI environment variable inside .env.local"
);
}
if (!database) {
throw new Error(
"Please define the DATABASE environment variable inside .env.local"
);
}
if (cachedConnection) return cachedConnection;
cachedConnection = await MongoClient.connect(mongoUri, {
useNewUrlParser: true,
useUnifiedTopology: true,
}).then((client) => ({
client,
db: client.db(database),
}));
return cachedConnection!;
}
Everytime I need to connect to MongoDB I do as follows:
const { db } = await connectToDatabase(config.URI, config.USERS_DATABASE);
const myUniversity = await db
.collection(config.MY_COLLECTION)
.findOne({})
Everything seems ok, so what is the problem?
The problem is that the connections to my DB don't close after I use them. In fact I thought that my server is stateless so after every time i use my DB, the connections end. But it is not true! They stay alive, and after few hours of using my app mongo atlas sends me an email saying that the limit is exceeded.
As you can see in this screenshot, this chart is ever growing. That means that connections stay on and they accumulate. How do you think I can solve this problem?
Keep in mind that it uses cachedConnection only if I use the same connection. If I call a different API from the first one it creates another connection and it doesn't enter in if (cachedConnection) block, but it goes forward till the end.
You can try this simple demo which will allow you to use the same connection throughout the application in different modules. There are three modules: the index.js is the starter program, the dbaccess.js is where you have code to create and maintain a connection which can be used again and again, and a apis.js module where you use the database connection to retrieve data.
index.js:
const express = require('express');
const mongo = require('./dbaccess');
const apis = require('./apis');
const app = express();
const init = async () => {
await mongo.connect();
app.listen(3000);
apis(app, mongo);
};
init();
dbaccess.js:
const { MongoClient } = require('mongodb');
class Mongo {
constructor() {
this.client = new MongoClient("mongodb://127.0.0.1:27017/", {
useNewUrlParser: true,
useUnifiedTopology: true
});
}
async connect() {
await this.client.connect();
console.log('Connected to MongoDB server.');
this.db = this.client.db('test');
console.log('Database:', this.db.databaseName);
}
}
module.exports = new Mongo();
apis.js:
module.exports = function(app, mongo) {
app.get('/', function(req, res) {
mongo.db.collection('users').find().limit(1).toArray(function(err, result) {
res.send('Doc: ' + JSON.stringify(result));
});
});
}
Change the appropriate values in the url, database name and collection name before trying.
Related
I have an Express app that was created with express generator. I have a standard app.js file that exports app. I also have a standard www file that imports app and is a starting point of the application:
const app = require('../app')
const debug = require('debug')('img-final:server')
const http = require('http')
const Mongo = require('../utils/dbConnection/dbConnection')
const port = normalizePort(process.env.PORT || '3000')
app.set('port', port)
/**
* Create HTTP server.
*/
const server = http.createServer(app)
/**
* Listen on provided port, on all network interfaces.
*/
async function startServer() {
try {
await Mongo.init()
debug('Connected correctly to DB')
server.listen(port)
} catch (err) {
debug(err)
}
}
startServer()
//some more unrelated code.
I also have a utility file for connecting to db dbConnection.js:
const MongoClient = require('mongodb').MongoClient
class Mongo {
async init() {
const client = new MongoClient(`mongodb://localhost:27017/img-new`, {
useNewUrlParser: true,
useUnifiedTopology: true,
})
await client.connect()
this.db = client.db('img-new')
}
getConnection() {
return this.db
}
}
module.exports = new Mongo()
My problem is that when I start my app const app = require('../app') is obviously running first, and wherever in my app route controllers I use getConnection(), the connection is undefined at that point because my Mongo.init() is running after const app = require('../app').
I'm trying to understand how to solve it in sane way. I guess I can move all require's and all other code inside startServer after await Mongo.init() , but it seems like there should be a better solution. Thank you.
Edit:
Is this an OK solution ?
const debug = require('debug')('img-final:server')
const http = require('http')
const Mongo = require('../utils/dbConnection/dbConnection')
async function startServer() {
try {
await Mongo.init()
const app = require('../app')
const port = normalizePort(process.env.PORT || '3000')
app.set('port', port)
const server = http.createServer(app)
server.listen(port)
} catch (err) {
debug(err)
}
}
startServer()
I have 1 solution but I'm not sure it satisfies your expectation.
In the getConenction method you check if this.db is undefined. If it's a case, just call init() method then return this.db. If not, you return this.db directly.
The code is like this :
async getConnection() {
if(!this.db) {
// connection to db is not established yet, we call `init` method
await this.init();
}
// this.db is defined here, we return the connection
return this.db;
}
And you don't have to call await Mongo.init() in the startServer() function
The previous answer by Đăng Khoa Đinh is the right direction. I add a bit of defensive coding to prevent multiple this.init() being called at the same time. Note: I did not code against errors while connecting.
const MongoClient = require('mongodb').MongoClient
class Mongo {
init() {
// Gerard we set isConnected to a promise
this.isConnected = new Promise(async (resolve, reject) => {
const client = new MongoClient(`mongodb://localhost:27017/img-new`, {
useNewUrlParser: true,
useUnifiedTopology: true,
})
await client.connect()
this.db = client.db('img-new')
resolve();
});
}
isConnected = null;
async getConnection() {
if(this.isConnected === null) {
// connection to db is not established yet, we call `init` method
this.init();
}
// Now either the DB is already connected, or a connection is in progress. We wait.
await this.isConnected;
// this.db is defined here, we return the connection
return this.db;
}
}
module.exports = new Mongo()
The caller will then just do
connection = await Mongo.getConnection();
So, I have read other topics here in StackOverflow that try to touch on this but don't have a clear solution to the problem.
First I created a config file for the mongo client which is exported from it.
const { MongoClient } = require('mongodb');
const authMechanism = 'SCRAM-SHA-1';
const user = encodeURIComponent(process.env.MONGODB_USER);
const password = encodeURIComponent(process.env.MONGODB_PASSWORD);
const uri = `mongodb://${user}:${password}#${process.env.MONGODB_URI}?authMechanism=${authMechanism}`;
const client = new MongoClient(uri, {
useUnifiedTopology: true,
useNewUrlParser: true,
loggerLevel: 'info',
});
module.exports = client;
From there I understand that you must have mongo client initialised once, and only once, before you listen to your application, hence I have created this index.js (entry point to the app) file that does that requiring the typical app.js where all the node config is.
const app = require('./app');
const db = require('../configs/db/db-config');
const port = process.env.PORT;
db.connect((err, client) => {
if (err) {
throw err;
}
const database = client.db('dbnamegoeshere');
app.listen(port, () => console.log(`Listening on port ${port}...`));
});
Now, in order for me to reuse that db anywhere I want to make queries or whatever, what is the best practice? How could I add it globally? would adding it globally affect the performance or be a bad practice?
I have seen other examples where people perform these two tasks using a class but yet again all in the same file, not with an export or a global.
One final question, where, and why should I close the db client connection.
Thank you.
I think adding the db connection object to the global object works perfectly. And if you are worried about performance, just reassign global vars to local ones.
//dbconnection.js
const debug = require('debug')('someapp:mongo');
const mongoClient = require('mongodb').MongoClient;
const mongoOptions = {};
const mongoUrl = process.env.MONGO_URL || "mongodb://localhost:27017/dbname";
function callback(err, r){
debug("callback: ", err, r);
};
module.exports = function () {
mongoClient.connect(mongoUrl, mongoOptions, (err, client) => {
if(err){
debug("MongoDB connection error: ", err);
throw err;
};
const db = global.db = client.db();
});
};
And just use it like so in your root application file, once added to the file the db connection should be globally available in other files in your project.
//server.js
require("./dbconnection")();
const userDB = global.db.collection("Users");
userDB.find({}).toArray((err, items)=>{});
I have hard time understanding why my code doesn't work. I am using node package mssql and want to have database pool connection initiation in separate file:
databaseConnection.js:
const sql = require("mssql/msnodesqlv8");
config = {
database: process.env.DB_NAME,
server: process.env.DB_SERVER,
driver: "msnodesqlv8",
options: {
trustedConnection: true
}
};
let pool = sql.connect(config);
module.exports = pool;
Then I have my express route file data.js
const express = require("express");
const router = express.Router();
const db = require("../configs/databaseConnection");
router.get("/dataList", async (req, res) => {
let allData = await db.request().query("select * from dataList");
console.log(allData);
res.render("dataList", { title: "Data list" });
});
module.exports = router;
However, when I start the server and go to the route I get error:
(node:13760) UnhandledPromiseRejectionWarning: TypeError: db.request is not a function
The thing is if I setup precisely as this example mssql documentation (where verything would be done in the route) it works. However, if database connection is in separate file it doesn't work.
I would appreciate any help understanding this
Regards,
Rokas
sql.connect returns a promise, so once we know that, we can either do a .then(result => ... or use await, for example:
If you want to store the db object at startup for later I'd suggest changing the line:
const db = require("../configs/databaseConnection");
to
let db = null;
require("../configs/databaseConnection").then(pool => {
db = pool;
});
I'm kinda new to module creation and was wondering about module.exports and waiting for async functions (like a mongo connect function for example) to complete and exporting the result. The variables get properly defined using async/await in the module, but when trying to log them by requiring the module, they show up as undefined. If someone could point me in the right direction, that'd be great. Here's the code I've got so far:
// module.js
const MongoClient = require('mongodb').MongoClient
const mongo_host = '127.0.0.1'
const mongo_db = 'test'
const mongo_port = '27017';
(async module => {
var client, db
var url = `mongodb://${mongo_host}:${mongo_port}/${mongo_db}`
try {
// Use connect method to connect to the Server
client = await MongoClient.connect(url, {
useNewUrlParser: true
})
db = client.db(mongo_db)
} catch (err) {
console.error(err)
} finally {
// Exporting mongo just to test things
console.log(client) // Just to test things I tried logging the client here and it works. It doesn't show 'undefined' like test.js does when trying to console.log it from there
module.exports = {
client,
db
}
}
})(module)
And here's the js that requires the module
// test.js
const {client} = require('./module')
console.log(client) // Logs 'undefined'
I'm fairly familiar with js and am still actively learning and looking into things like async/await and like features, but yeah... I can't really figure that one out
You have to export synchronously, so its impossible to export client and db directly. However you could export a Promise that resolves to client and db:
module.exports = (async function() {
const client = await MongoClient.connect(url, {
useNewUrlParser: true
});
const db = client.db(mongo_db);
return { client, db };
})();
So then you can import it as:
const {client, db} = await require("yourmodule");
(that has to be in an async function itself)
PS: console.error(err) is not a proper error handler, if you cant handle the error just crash
the solution provided above by #Jonas Wilms is working but requires to call requires in an async function each time we want to reuse the connection. an alternative way is to use a callback function to return the mongoDB client object.
mongo.js:
const MongoClient = require('mongodb').MongoClient;
const uri = "mongodb+srv://<user>:<pwd>#<host and port>?retryWrites=true";
const mongoClient = async function(cb) {
const client = await MongoClient.connect(uri, {
useNewUrlParser: true
});
cb(client);
};
module.exports = {mongoClient}
then we can use mongoClient method in a diffrent file(express route or any other js file).
app.js:
var client;
const mongo = require('path to mongo.js');
mongo.mongoClient((connection) => {
client = connection;
});
//declare express app and listen....
//simple post reuest to store a student..
app.post('/', async (req, res, next) => {
const newStudent = {
name: req.body.name,
description: req.body.description,
studentId: req.body.studetId,
image: req.body.image
};
try
{
await client.db('university').collection('students').insertOne({newStudent});
}
catch(err)
{
console.log(err);
return res.status(500).json({ error: err});
}
return res.status(201).json({ message: 'Student added'});
};
I am working on a Koa + Mongodb backend. My question is: When should I close the db, or does Mongodb manage that because I am not closing any of them right now and it seems fine.
// app.js
const Koa = require('koa')
const database = require('./database')
const app = new Koa()
database
.connet()
.then(() => {app.listen(':8080')})
.catch((err) => {console.error(err)})
// ./database.js
const MongoClient = require('mongodb').MongoClient
const Model = require('./model')
class Database {
async connect() {
if (!db) {
db = await MongoClient.connect("localhost:27017")
this.item = new Model(db, 'item_collection')
}
}
}
module.exports = new Database()
// ./model.js
class Model {
constructor(db, collectionName) {
this.name = collectionName
this.database = database
}
async findAll() {
const result = await this.db.collection(this.name).find().toArray()
if (!result) {
throw new Error('error')
}
return result
}
}
module.exports = Model
I also ran a stress test using vegeta to make API request to the server at 100 request / second and the response time is good. So, am I worried about premature optimization here? If not, when should I close the db?
As Koa keeps running (and in your case listening on port 8080) you should not close the db connection.
If you are running scripts that are expected to end (tasks running on cron, etc) you should manually close the connection when you are finished with all of your db tasks.
You can take a look at this example for express.js (Koa's sister framework)