I'm kinda new to module creation and was wondering about module.exports and waiting for async functions (like a mongo connect function for example) to complete and exporting the result. The variables get properly defined using async/await in the module, but when trying to log them by requiring the module, they show up as undefined. If someone could point me in the right direction, that'd be great. Here's the code I've got so far:
// module.js
const MongoClient = require('mongodb').MongoClient
const mongo_host = '127.0.0.1'
const mongo_db = 'test'
const mongo_port = '27017';
(async module => {
var client, db
var url = `mongodb://${mongo_host}:${mongo_port}/${mongo_db}`
try {
// Use connect method to connect to the Server
client = await MongoClient.connect(url, {
useNewUrlParser: true
})
db = client.db(mongo_db)
} catch (err) {
console.error(err)
} finally {
// Exporting mongo just to test things
console.log(client) // Just to test things I tried logging the client here and it works. It doesn't show 'undefined' like test.js does when trying to console.log it from there
module.exports = {
client,
db
}
}
})(module)
And here's the js that requires the module
// test.js
const {client} = require('./module')
console.log(client) // Logs 'undefined'
I'm fairly familiar with js and am still actively learning and looking into things like async/await and like features, but yeah... I can't really figure that one out
You have to export synchronously, so its impossible to export client and db directly. However you could export a Promise that resolves to client and db:
module.exports = (async function() {
const client = await MongoClient.connect(url, {
useNewUrlParser: true
});
const db = client.db(mongo_db);
return { client, db };
})();
So then you can import it as:
const {client, db} = await require("yourmodule");
(that has to be in an async function itself)
PS: console.error(err) is not a proper error handler, if you cant handle the error just crash
the solution provided above by #Jonas Wilms is working but requires to call requires in an async function each time we want to reuse the connection. an alternative way is to use a callback function to return the mongoDB client object.
mongo.js:
const MongoClient = require('mongodb').MongoClient;
const uri = "mongodb+srv://<user>:<pwd>#<host and port>?retryWrites=true";
const mongoClient = async function(cb) {
const client = await MongoClient.connect(uri, {
useNewUrlParser: true
});
cb(client);
};
module.exports = {mongoClient}
then we can use mongoClient method in a diffrent file(express route or any other js file).
app.js:
var client;
const mongo = require('path to mongo.js');
mongo.mongoClient((connection) => {
client = connection;
});
//declare express app and listen....
//simple post reuest to store a student..
app.post('/', async (req, res, next) => {
const newStudent = {
name: req.body.name,
description: req.body.description,
studentId: req.body.studetId,
image: req.body.image
};
try
{
await client.db('university').collection('students').insertOne({newStudent});
}
catch(err)
{
console.log(err);
return res.status(500).json({ error: err});
}
return res.status(201).json({ message: 'Student added'});
};
Related
I have a problem with the approach I use to connect to Mondo DB.
I use the following method:
import { Db, MongoClient } from "mongodb";
let cachedConnection: { client: MongoClient; db: Db } | null = null;
export async function connectToDatabase(mongoUri?: string, database?: string) {
if (!mongoUri) {
throw new Error(
"Please define the MONGO_URI environment variable inside .env.local"
);
}
if (!database) {
throw new Error(
"Please define the DATABASE environment variable inside .env.local"
);
}
if (cachedConnection) return cachedConnection;
cachedConnection = await MongoClient.connect(mongoUri, {
useNewUrlParser: true,
useUnifiedTopology: true,
}).then((client) => ({
client,
db: client.db(database),
}));
return cachedConnection!;
}
Everytime I need to connect to MongoDB I do as follows:
const { db } = await connectToDatabase(config.URI, config.USERS_DATABASE);
const myUniversity = await db
.collection(config.MY_COLLECTION)
.findOne({})
Everything seems ok, so what is the problem?
The problem is that the connections to my DB don't close after I use them. In fact I thought that my server is stateless so after every time i use my DB, the connections end. But it is not true! They stay alive, and after few hours of using my app mongo atlas sends me an email saying that the limit is exceeded.
As you can see in this screenshot, this chart is ever growing. That means that connections stay on and they accumulate. How do you think I can solve this problem?
Keep in mind that it uses cachedConnection only if I use the same connection. If I call a different API from the first one it creates another connection and it doesn't enter in if (cachedConnection) block, but it goes forward till the end.
You can try this simple demo which will allow you to use the same connection throughout the application in different modules. There are three modules: the index.js is the starter program, the dbaccess.js is where you have code to create and maintain a connection which can be used again and again, and a apis.js module where you use the database connection to retrieve data.
index.js:
const express = require('express');
const mongo = require('./dbaccess');
const apis = require('./apis');
const app = express();
const init = async () => {
await mongo.connect();
app.listen(3000);
apis(app, mongo);
};
init();
dbaccess.js:
const { MongoClient } = require('mongodb');
class Mongo {
constructor() {
this.client = new MongoClient("mongodb://127.0.0.1:27017/", {
useNewUrlParser: true,
useUnifiedTopology: true
});
}
async connect() {
await this.client.connect();
console.log('Connected to MongoDB server.');
this.db = this.client.db('test');
console.log('Database:', this.db.databaseName);
}
}
module.exports = new Mongo();
apis.js:
module.exports = function(app, mongo) {
app.get('/', function(req, res) {
mongo.db.collection('users').find().limit(1).toArray(function(err, result) {
res.send('Doc: ' + JSON.stringify(result));
});
});
}
Change the appropriate values in the url, database name and collection name before trying.
I have an Express app that was created with express generator. I have a standard app.js file that exports app. I also have a standard www file that imports app and is a starting point of the application:
const app = require('../app')
const debug = require('debug')('img-final:server')
const http = require('http')
const Mongo = require('../utils/dbConnection/dbConnection')
const port = normalizePort(process.env.PORT || '3000')
app.set('port', port)
/**
* Create HTTP server.
*/
const server = http.createServer(app)
/**
* Listen on provided port, on all network interfaces.
*/
async function startServer() {
try {
await Mongo.init()
debug('Connected correctly to DB')
server.listen(port)
} catch (err) {
debug(err)
}
}
startServer()
//some more unrelated code.
I also have a utility file for connecting to db dbConnection.js:
const MongoClient = require('mongodb').MongoClient
class Mongo {
async init() {
const client = new MongoClient(`mongodb://localhost:27017/img-new`, {
useNewUrlParser: true,
useUnifiedTopology: true,
})
await client.connect()
this.db = client.db('img-new')
}
getConnection() {
return this.db
}
}
module.exports = new Mongo()
My problem is that when I start my app const app = require('../app') is obviously running first, and wherever in my app route controllers I use getConnection(), the connection is undefined at that point because my Mongo.init() is running after const app = require('../app').
I'm trying to understand how to solve it in sane way. I guess I can move all require's and all other code inside startServer after await Mongo.init() , but it seems like there should be a better solution. Thank you.
Edit:
Is this an OK solution ?
const debug = require('debug')('img-final:server')
const http = require('http')
const Mongo = require('../utils/dbConnection/dbConnection')
async function startServer() {
try {
await Mongo.init()
const app = require('../app')
const port = normalizePort(process.env.PORT || '3000')
app.set('port', port)
const server = http.createServer(app)
server.listen(port)
} catch (err) {
debug(err)
}
}
startServer()
I have 1 solution but I'm not sure it satisfies your expectation.
In the getConenction method you check if this.db is undefined. If it's a case, just call init() method then return this.db. If not, you return this.db directly.
The code is like this :
async getConnection() {
if(!this.db) {
// connection to db is not established yet, we call `init` method
await this.init();
}
// this.db is defined here, we return the connection
return this.db;
}
And you don't have to call await Mongo.init() in the startServer() function
The previous answer by Đăng Khoa Đinh is the right direction. I add a bit of defensive coding to prevent multiple this.init() being called at the same time. Note: I did not code against errors while connecting.
const MongoClient = require('mongodb').MongoClient
class Mongo {
init() {
// Gerard we set isConnected to a promise
this.isConnected = new Promise(async (resolve, reject) => {
const client = new MongoClient(`mongodb://localhost:27017/img-new`, {
useNewUrlParser: true,
useUnifiedTopology: true,
})
await client.connect()
this.db = client.db('img-new')
resolve();
});
}
isConnected = null;
async getConnection() {
if(this.isConnected === null) {
// connection to db is not established yet, we call `init` method
this.init();
}
// Now either the DB is already connected, or a connection is in progress. We wait.
await this.isConnected;
// this.db is defined here, we return the connection
return this.db;
}
}
module.exports = new Mongo()
The caller will then just do
connection = await Mongo.getConnection();
So, I have read other topics here in StackOverflow that try to touch on this but don't have a clear solution to the problem.
First I created a config file for the mongo client which is exported from it.
const { MongoClient } = require('mongodb');
const authMechanism = 'SCRAM-SHA-1';
const user = encodeURIComponent(process.env.MONGODB_USER);
const password = encodeURIComponent(process.env.MONGODB_PASSWORD);
const uri = `mongodb://${user}:${password}#${process.env.MONGODB_URI}?authMechanism=${authMechanism}`;
const client = new MongoClient(uri, {
useUnifiedTopology: true,
useNewUrlParser: true,
loggerLevel: 'info',
});
module.exports = client;
From there I understand that you must have mongo client initialised once, and only once, before you listen to your application, hence I have created this index.js (entry point to the app) file that does that requiring the typical app.js where all the node config is.
const app = require('./app');
const db = require('../configs/db/db-config');
const port = process.env.PORT;
db.connect((err, client) => {
if (err) {
throw err;
}
const database = client.db('dbnamegoeshere');
app.listen(port, () => console.log(`Listening on port ${port}...`));
});
Now, in order for me to reuse that db anywhere I want to make queries or whatever, what is the best practice? How could I add it globally? would adding it globally affect the performance or be a bad practice?
I have seen other examples where people perform these two tasks using a class but yet again all in the same file, not with an export or a global.
One final question, where, and why should I close the db client connection.
Thank you.
I think adding the db connection object to the global object works perfectly. And if you are worried about performance, just reassign global vars to local ones.
//dbconnection.js
const debug = require('debug')('someapp:mongo');
const mongoClient = require('mongodb').MongoClient;
const mongoOptions = {};
const mongoUrl = process.env.MONGO_URL || "mongodb://localhost:27017/dbname";
function callback(err, r){
debug("callback: ", err, r);
};
module.exports = function () {
mongoClient.connect(mongoUrl, mongoOptions, (err, client) => {
if(err){
debug("MongoDB connection error: ", err);
throw err;
};
const db = global.db = client.db();
});
};
And just use it like so in your root application file, once added to the file the db connection should be globally available in other files in your project.
//server.js
require("./dbconnection")();
const userDB = global.db.collection("Users");
userDB.find({}).toArray((err, items)=>{});
I am using Node and Express with the the mssql npm package to connect to an SQL Server database. I do this in my app.js file which sets up a global variable to create a connectionPool to the database like so (I omitted some boilerplate stuff for brevity):
const mssql = require('mssql/msnodesqlv8'); // needs to use msnodesqlv8 driver for Windows Authentication to work
const express = require('express');
const app = express();
const DB_MYDB_Config = {
server: "localhost",
database: "MYDB",
options: {
trustedConnection: true // Windows auth enabled hence no username/password required
}
};
global.MSSQL_MYDB = new mssql.ConnectionPool(DB_MYDB_Config); //connectionPool available to whole app
I have a Model file called offer.js which just does this:
async function getOffersAll() {
await MSSQL_MYDB.connect(); // connects to the database
try {
var result = await MSSQL_MYDB.request(MSSQL_MYDB).query('SELECT Top(1) * FROM [dbo].[Offer]');
return result; // returns the recordset of data
} catch (error) {
console.log(error);
} finally {
if (MSSQL_MYDB){
try {
await MSSQL_MYDB.close(); // closes the DB connection
}
catch (error) {
console.log('Error closing connection');
}
}
}
}
exports.getOffersAll = getOffersAll;
So far so good. I then have a Controller file index.js which doesn't really work (explained with comments):
var router = require('express').Router();
const Offer = require('../models/offer'); // the `offer.js` file
/* the function below works perfectly */
(async function () {
var rsOffersAll = await Offer.getOffersAll();
console.dir(rsOffersAll); // works great! recordset rsOffersAll is printed to console
})();
/* this is where it goes wrong even after commenting out the test function above */
router.get('/', async function(req, res) {
var rsOffersAll = await Offer.getOffersAll(); // this just hangs and eventually I get a login failed error for SQL Server.
console.dir(rsOffersAll);
res.render('index', { pagetitle: 'Homepage'}); // never renders
});
module.exports = router;
My question is why does the first async function() that awaits a result from Offer.getOffersAll() not fail, but the same async function placed within the router.get('/') handler fails with a login error? If I remove the var rsOffersAll = await Offer.getOffersAll(); from the router.get('/') handler then the page renders, but of course I have no data to pass to it.
The exact same thing happens even if I store the test function's value in a variable and try to put it in the router.get() handler like this:
async function getOffersTest() {
return await Offer.getOffersAll();
}
router.get('/', async function(req, res) {
var rsOffersAll = await getOffersTest(); // still breaks
console.dir(rsOffersAll);
res.render('index', { pagetitle: 'Homepage'}); // still never renders
});
My ultimate question how do I fix this so it just works the way it should in that when the homepage is visited, the router waits for the data to be returned from the database and then I can pass it to my view or just print to the console if I want?
because of this line global.MSSQL_MYDB = new mssql.ConnectionPool(DB_MYDB_Config);,
when you execute this code outside of router,
(async function () {
var rsOffersAll = await Offer.getOffersAll();
console.dir(rsOffersAll); // works great! recordset rsOffersAll is printed to console
})();
getOffersAll has access to global variable,
and you can successfully connect with db in line await MSSQL_MYDB.connect(); //
but as for router, global scope is the current module.exports object, not the global object.
Solution
you can set MSSQL_MYDB in app like this,
app.set('MSSQL_MYDB')
then you can get this same variable in following function like this
router.get('/', async function(req, res) {
const MSSQL_MYDB = req.app.get('MSSQL_MYDB')
var rsOffersAll = await getOffersTest(MSSQL_MYDB );
console.dir(rsOffersAll);
res.render('index', { pagetitle: 'Homepage'}); // still never renders
});
This whole problem was just solved and it is a bug or something in node mssql package. It only works if you provide a username and password. If you use options: {trustedConnection: true } to enable windows authentication, then the router can never log in. I don't know why this is the case, but if you just supply a username and password (I used sa for testing) then it works fine.
I have hard time understanding why my code doesn't work. I am using node package mssql and want to have database pool connection initiation in separate file:
databaseConnection.js:
const sql = require("mssql/msnodesqlv8");
config = {
database: process.env.DB_NAME,
server: process.env.DB_SERVER,
driver: "msnodesqlv8",
options: {
trustedConnection: true
}
};
let pool = sql.connect(config);
module.exports = pool;
Then I have my express route file data.js
const express = require("express");
const router = express.Router();
const db = require("../configs/databaseConnection");
router.get("/dataList", async (req, res) => {
let allData = await db.request().query("select * from dataList");
console.log(allData);
res.render("dataList", { title: "Data list" });
});
module.exports = router;
However, when I start the server and go to the route I get error:
(node:13760) UnhandledPromiseRejectionWarning: TypeError: db.request is not a function
The thing is if I setup precisely as this example mssql documentation (where verything would be done in the route) it works. However, if database connection is in separate file it doesn't work.
I would appreciate any help understanding this
Regards,
Rokas
sql.connect returns a promise, so once we know that, we can either do a .then(result => ... or use await, for example:
If you want to store the db object at startup for later I'd suggest changing the line:
const db = require("../configs/databaseConnection");
to
let db = null;
require("../configs/databaseConnection").then(pool => {
db = pool;
});