We Keep Coding

How to store data from a MariaDB in my node.js environment

I have a MariaDB that stores Energy-Data like voltage, frequency and so on. My aim is to visualize the data in a Web-application. Though i achieved to connect the MariaDB to node.js and log the data on a specific port thanks to the code below, i don't have a clue how to store this data for further mathematic operations or visualizations.
How can i store the data for further operations?
const express = require('express');
const pool = require('./db');
const app = express();
const port = 4999;
// expose an endpoint "persons"
app.get('/persons', async (req, res) => {
let conn;
try {
// make a connection to MariaDB
conn = await pool.getConnection();
// create a new query to fetch all records from the table
var query = "select * from Herget_Netz2_WirkleistungL1";
// run the query and set the result to a new variable
var rows = await conn.query(query);
console.log('Daten kommen');
// return the results
res.send(rows);
} catch (err) {
throw err;
} finally {
if (conn) return conn.release();
}
});
app.listen(port, () => console.log(`Listening on pfort ${port}`));

This question is quite broad.
It sounds like you need to set up a frontend and call fetch on your endpoint, something like:
fetch(<your-url>/persons)
.then(r => r.json())
.then(yourData => "<p>" + yourData "</p>")
Your data will be interpolated into HTML then. You will need to iterate over it.
The "storage" will take place in the variable you define in the second .then(yourData) of the promise for you to do further operations on.
You should search for tutorials like "set up frontend with maria db database and node backend".

Node.Js MSSQL Query Timeout Expired

I am using Node Express API to run SQL queries to populate a dashboard of data. I am using the mssql-node package to do so. Sometimes it runs flawlessly, other times I get the following error:
[Error: [Microsoft][SQL Server Native Client 11.0]Query timeout expired]
I am creating a poolPromise with a connectionPool to the db, then I pass that object to my other controllers which run the specific queries to populate data. I run the server which initiates the db.js script and connects to MSSQL with a pool connection.
db.js:
// for connecting to sql server
const sql = require('mssql/msnodesqlv8');
// db config to connect via windows auth
const dbConfig = {
driver: 'msnodesqlv8',
connectionString: 'Driver={SQL Server Native Client 11.0};Server={my_server};Database={my_db};Trusted_Connection={yes};',
pool: {
idleTimeoutMillis: 60000
}
};
// create a connectionpool object to pass to controllers
// this should keep a sql connection open indefinitely that we can query when the server is running
const poolPromise = new sql.ConnectionPool(dbConfig)
.connect()
.then(pool => {
console.log('Connected to MSSQL');
return pool;
})
.catch(err => console.log('Database Connection Failed! Bad Config: ', err))
module.exports = { sql, poolPromise };
An example of one of my controllers and how I use the poolPromise object is below. I currently have about 7 of these controllers that run their own specific query to populate a specific element on the dashboard. The performance of the queries each run within 1-10 seconds (depending on current server load, as I am querying an enterprise production server/db, this can vary). As I mentioned earlier, the queries run flawlessly sometimes and I have no issues, but at other times I do have issues. Is this a symptom of me querying from a shared production server? Is it preferred to query from a server that has less load? Or am I doing something in my code that could be improved?
const { sql, poolPromise } = require('../db');
// function to get data
const getData = async (req, res) => {
try {
// create query parameters from user request
let id= req.query.id;
// create query from connectionPool
let pool = await poolPromise;
let qry = `
select * from tbl where id = #Id
`
let data = await pool.request()
.input('Id', sql.VarChar(sql.MAX), id)
.query(qry);
// send 200 status and return records
res.status(200);
res.send(data.recordset);
} catch(err) {
console.log('Error:');
console.log(err);
res.sendStatus(500);
}
};
module.exports = { getData };

Mongo DB problem - connections accumulation

I have a problem with the approach I use to connect to Mondo DB.
I use the following method:
import { Db, MongoClient } from "mongodb";
let cachedConnection: { client: MongoClient; db: Db } | null = null;
export async function connectToDatabase(mongoUri?: string, database?: string) {
if (!mongoUri) {
throw new Error(
"Please define the MONGO_URI environment variable inside .env.local"
);
}
if (!database) {
throw new Error(
"Please define the DATABASE environment variable inside .env.local"
);
}
if (cachedConnection) return cachedConnection;
cachedConnection = await MongoClient.connect(mongoUri, {
useNewUrlParser: true,
useUnifiedTopology: true,
}).then((client) => ({
client,
db: client.db(database),
}));
return cachedConnection!;
}
Everytime I need to connect to MongoDB I do as follows:
const { db } = await connectToDatabase(config.URI, config.USERS_DATABASE);
const myUniversity = await db
.collection(config.MY_COLLECTION)
.findOne({})
Everything seems ok, so what is the problem?
The problem is that the connections to my DB don't close after I use them. In fact I thought that my server is stateless so after every time i use my DB, the connections end. But it is not true! They stay alive, and after few hours of using my app mongo atlas sends me an email saying that the limit is exceeded.
As you can see in this screenshot, this chart is ever growing. That means that connections stay on and they accumulate. How do you think I can solve this problem?
Keep in mind that it uses cachedConnection only if I use the same connection. If I call a different API from the first one it creates another connection and it doesn't enter in if (cachedConnection) block, but it goes forward till the end.

You can try this simple demo which will allow you to use the same connection throughout the application in different modules. There are three modules: the index.js is the starter program, the dbaccess.js is where you have code to create and maintain a connection which can be used again and again, and a apis.js module where you use the database connection to retrieve data.
index.js:
const express = require('express');
const mongo = require('./dbaccess');
const apis = require('./apis');
const app = express();
const init = async () => {
await mongo.connect();
app.listen(3000);
apis(app, mongo);
};
init();
dbaccess.js:
const { MongoClient } = require('mongodb');
class Mongo {
constructor() {
this.client = new MongoClient("mongodb://127.0.0.1:27017/", {
useNewUrlParser: true,
useUnifiedTopology: true
});
}
async connect() {
await this.client.connect();
console.log('Connected to MongoDB server.');
this.db = this.client.db('test');
console.log('Database:', this.db.databaseName);
}
}
module.exports = new Mongo();
apis.js:
module.exports = function(app, mongo) {
app.get('/', function(req, res) {
mongo.db.collection('users').find().limit(1).toArray(function(err, result) {
res.send('Doc: ' + JSON.stringify(result));
});
});
}
Change the appropriate values in the url, database name and collection name before trying.

How do I solve a problem with my DB connection and require?

I have an Express app that was created with express generator. I have a standard app.js file that exports app. I also have a standard www file that imports app and is a starting point of the application:
const app = require('../app')
const debug = require('debug')('img-final:server')
const http = require('http')
const Mongo = require('../utils/dbConnection/dbConnection')
const port = normalizePort(process.env.PORT || '3000')
app.set('port', port)
/**
* Create HTTP server.
*/
const server = http.createServer(app)
/**
* Listen on provided port, on all network interfaces.
*/
async function startServer() {
try {
await Mongo.init()
debug('Connected correctly to DB')
server.listen(port)
} catch (err) {
debug(err)
}
}
startServer()
//some more unrelated code.
I also have a utility file for connecting to db dbConnection.js:
const MongoClient = require('mongodb').MongoClient
class Mongo {
async init() {
const client = new MongoClient(`mongodb://localhost:27017/img-new`, {
useNewUrlParser: true,
useUnifiedTopology: true,
})
await client.connect()
this.db = client.db('img-new')
}
getConnection() {
return this.db
}
}
module.exports = new Mongo()
My problem is that when I start my app const app = require('../app') is obviously running first, and wherever in my app route controllers I use getConnection(), the connection is undefined at that point because my Mongo.init() is running after const app = require('../app').
I'm trying to understand how to solve it in sane way. I guess I can move all require's and all other code inside startServer after await Mongo.init() , but it seems like there should be a better solution. Thank you.
Edit:
Is this an OK solution ?
const debug = require('debug')('img-final:server')
const http = require('http')
const Mongo = require('../utils/dbConnection/dbConnection')
async function startServer() {
try {
await Mongo.init()
const app = require('../app')
const port = normalizePort(process.env.PORT || '3000')
app.set('port', port)
const server = http.createServer(app)
server.listen(port)
} catch (err) {
debug(err)
}
}
startServer()

I have 1 solution but I'm not sure it satisfies your expectation.
In the getConenction method you check if this.db is undefined. If it's a case, just call init() method then return this.db. If not, you return this.db directly.
The code is like this :
async getConnection() {
if(!this.db) {
// connection to db is not established yet, we call `init` method
await this.init();
}
// this.db is defined here, we return the connection
return this.db;
}
And you don't have to call await Mongo.init() in the startServer() function

The previous answer by Đăng Khoa Đinh is the right direction. I add a bit of defensive coding to prevent multiple this.init() being called at the same time. Note: I did not code against errors while connecting.
const MongoClient = require('mongodb').MongoClient
class Mongo {
init() {
// Gerard we set isConnected to a promise
this.isConnected = new Promise(async (resolve, reject) => {
const client = new MongoClient(`mongodb://localhost:27017/img-new`, {
useNewUrlParser: true,
useUnifiedTopology: true,
})
await client.connect()
this.db = client.db('img-new')
resolve();
});
}
isConnected = null;
async getConnection() {
if(this.isConnected === null) {
// connection to db is not established yet, we call `init` method
this.init();
}
// Now either the DB is already connected, or a connection is in progress. We wait.
await this.isConnected;
// this.db is defined here, we return the connection
return this.db;
}
}
module.exports = new Mongo()
The caller will then just do
connection = await Mongo.getConnection();

How to reuse mongo client using globals, best practices

So, I have read other topics here in StackOverflow that try to touch on this but don't have a clear solution to the problem.
First I created a config file for the mongo client which is exported from it.
const { MongoClient } = require('mongodb');
const authMechanism = 'SCRAM-SHA-1';
const user = encodeURIComponent(process.env.MONGODB_USER);
const password = encodeURIComponent(process.env.MONGODB_PASSWORD);
const uri = `mongodb://${user}:${password}#${process.env.MONGODB_URI}?authMechanism=${authMechanism}`;
const client = new MongoClient(uri, {
useUnifiedTopology: true,
useNewUrlParser: true,
loggerLevel: 'info',
});
module.exports = client;
From there I understand that you must have mongo client initialised once, and only once, before you listen to your application, hence I have created this index.js (entry point to the app) file that does that requiring the typical app.js where all the node config is.
const app = require('./app');
const db = require('../configs/db/db-config');
const port = process.env.PORT;
db.connect((err, client) => {
if (err) {
throw err;
}
const database = client.db('dbnamegoeshere');
app.listen(port, () => console.log(`Listening on port ${port}...`));
});
Now, in order for me to reuse that db anywhere I want to make queries or whatever, what is the best practice? How could I add it globally? would adding it globally affect the performance or be a bad practice?
I have seen other examples where people perform these two tasks using a class but yet again all in the same file, not with an export or a global.
One final question, where, and why should I close the db client connection.
Thank you.

I think adding the db connection object to the global object works perfectly. And if you are worried about performance, just reassign global vars to local ones.
//dbconnection.js
const debug = require('debug')('someapp:mongo');
const mongoClient = require('mongodb').MongoClient;
const mongoOptions = {};
const mongoUrl = process.env.MONGO_URL || "mongodb://localhost:27017/dbname";
function callback(err, r){
debug("callback: ", err, r);
};
module.exports = function () {
mongoClient.connect(mongoUrl, mongoOptions, (err, client) => {
if(err){
debug("MongoDB connection error: ", err);
throw err;
};
const db = global.db = client.db();
});
};
And just use it like so in your root application file, once added to the file the db connection should be globally available in other files in your project.
//server.js
require("./dbconnection")();
const userDB = global.db.collection("Users");
userDB.find({}).toArray((err, items)=>{});