We Keep Coding

Bind problem in SQL query in Node, Express, Mysql2 app

I have been following a tutorial on setting up REST APIs in Node, using Express for an app that accesses an existing MariaDB database. My version only needs to read data and I have the DB co-located with the Node application (same host).
My goal for this entry-level example is to just access the data, using static SQL, so I can see it rendered in the web page by the JSON pritifier.
[Next, I want to present the data in a table (EJS?). Later, when I can get that to work, I'll add form controls (React?) to let a user specify start and end date bounds for the SQL query. Finally I'll aim to render the data as a line graph (D3js).]
The tutorial runs the web server successfully (it returns 'OK' on the base URL), but when I go to URL/solarData it tries an async function to getMultiple rows from the DB, it responds:
Bind parameters must not contain undefined. To pass SQL NULL specify JS null TypeError: Bind parameters must not contain undefined. To pass SQL NULL specify JS null
at /SunnyData/solarViz/node_modules/mysql2/lib/connection.js:628:17
at Array.forEach (<anonymous>)
at Connection.execute (/SunnyData/solarViz/node_modules/mysql2/lib/connection.js:620:22)
at /SunnyData/solarViz/node_modules/mysql2/promise.js:120:11
at new Promise (<anonymous>)
at PromiseConnection.execute (/SunnyData/solarViz/node_modules/mysql2/promise.js:117:12)
at Object.query (/SunnyData/solarViz/services/db.js:6:40)
at processTicksAndRejections (internal/process/task_queues.js:95:5)
at async Object.getMultiple (/SunnyData/solarViz/services/solarData.js:7:16)
at async /SunnyData/solarViz/routes/solarData.js:8:14
app.js:61
./app.js
const express = require('express');
const app = express();
const port = process.env.PORT || 3800;
const solarDataRouter = require('./routes/solarData');
app.use(express.json());
app.use(
express.urlencoded({
extended: true,
})
);
app.get('/', (req, res) => {
res.json({'message': 'ok'});
})
app.use('/solarData', solarDataRouter);
/* Error handler middleware */
app.use((err, req, res, next) => {
const statusCode = err.statusCode || 500;
console.error(err.message, err.stack);
res.status(statusCode).json({'message': err.message});
return;
});
app.listen(port, () => {
console.log(`Example app listening at http://localhost:${port}`)
});
./routes/solarData.js
const express = require('express');
const router = express.Router();
const solarData = require('../services/solarData');
/* GET solar data. */
router.get('/', async function(req, res, next) {
try {
res.json(await solarData.getMultiple(req.query.page));
} catch (err) {
console.error(`Error while getting solar data `, err.message);
next(err);
}
});
module.exports = router;
./config.js
const env = process.env;
const config = {
db: {
host: env.SUNNY_HOST,
user: env.SUNNY_USER,
password: env.SUNNY_PW,
database: env.SUNNY_DB,
},
listPerPage: env.LIST_PER_PAGE,
};
module.exports = config;
./services/solarData.js
const db = require('./db');
const helper = require('../helper');
const config = require('../config');
async function getMultiple(page = 1){
const offset = helper.getOffset(page, config.listPerPage);
const rows = await db.query(
`SELECT * FROM DTP LIMIT ?,?`, [offset, config.listPerPage]
);
const data = helper.emptyOrRows(rows);
const meta = {page};
return {
data,
meta
}
}
module.exports.getMultiple = getMultiple;
./services/db.js
const mysql = require('mysql2/promise');
const config = require('../config');
async function query(sql, params) {
const connection = await mysql.createConnection(config.db);
const [results, ] = await connection.execute(sql, params);
return results;
}
module.exports = {
query
}
I've left out the ./helper.js
Everything runs fine until I direct the webpage to /solarData. At that point I get the Debug Console (vscode) mentioned up-front
Searching seems to point at a mysql2 shortcoming/bug but not at a practical solution
If you respond, please describe the 'bind' mechanism, as I'm not sure what's going on.
Hope I've put enough info in. Please ask if I need to add anything else.

The error says
Bind parameters must not contain undefined.
It means that in the file ./services/solarData.js on the line
const rows = await db.query(
`SELECT * FROM DTP LIMIT ?,?`, [offset, config.listPerPage]
);
Some of the 2 variables is undefined, you need to check offset and config.listPerPage to be defined.
Just use
console.log('offset: ' + offset)
console.log('listPerPage: ' + config.listPerPage)
and you will find out what is undefined in your case

Mongo DB problem - connections accumulation

I have a problem with the approach I use to connect to Mondo DB.
I use the following method:
import { Db, MongoClient } from "mongodb";
let cachedConnection: { client: MongoClient; db: Db } | null = null;
export async function connectToDatabase(mongoUri?: string, database?: string) {
if (!mongoUri) {
throw new Error(
"Please define the MONGO_URI environment variable inside .env.local"
);
}
if (!database) {
throw new Error(
"Please define the DATABASE environment variable inside .env.local"
);
}
if (cachedConnection) return cachedConnection;
cachedConnection = await MongoClient.connect(mongoUri, {
useNewUrlParser: true,
useUnifiedTopology: true,
}).then((client) => ({
client,
db: client.db(database),
}));
return cachedConnection!;
}
Everytime I need to connect to MongoDB I do as follows:
const { db } = await connectToDatabase(config.URI, config.USERS_DATABASE);
const myUniversity = await db
.collection(config.MY_COLLECTION)
.findOne({})
Everything seems ok, so what is the problem?
The problem is that the connections to my DB don't close after I use them. In fact I thought that my server is stateless so after every time i use my DB, the connections end. But it is not true! They stay alive, and after few hours of using my app mongo atlas sends me an email saying that the limit is exceeded.
As you can see in this screenshot, this chart is ever growing. That means that connections stay on and they accumulate. How do you think I can solve this problem?
Keep in mind that it uses cachedConnection only if I use the same connection. If I call a different API from the first one it creates another connection and it doesn't enter in if (cachedConnection) block, but it goes forward till the end.

You can try this simple demo which will allow you to use the same connection throughout the application in different modules. There are three modules: the index.js is the starter program, the dbaccess.js is where you have code to create and maintain a connection which can be used again and again, and a apis.js module where you use the database connection to retrieve data.
index.js:
const express = require('express');
const mongo = require('./dbaccess');
const apis = require('./apis');
const app = express();
const init = async () => {
await mongo.connect();
app.listen(3000);
apis(app, mongo);
};
init();
dbaccess.js:
const { MongoClient } = require('mongodb');
class Mongo {
constructor() {
this.client = new MongoClient("mongodb://127.0.0.1:27017/", {
useNewUrlParser: true,
useUnifiedTopology: true
});
}
async connect() {
await this.client.connect();
console.log('Connected to MongoDB server.');
this.db = this.client.db('test');
console.log('Database:', this.db.databaseName);
}
}
module.exports = new Mongo();
apis.js:
module.exports = function(app, mongo) {
app.get('/', function(req, res) {
mongo.db.collection('users').find().limit(1).toArray(function(err, result) {
res.send('Doc: ' + JSON.stringify(result));
});
});
}
Change the appropriate values in the url, database name and collection name before trying.

Node.js Database Module with Sequelize

To make my code more readable, I'm trying to move all database related code into a single file. and use Sequelize as ORM. I would like that this file, when included provide a ready to use Database. Tables schemas are also managed by Sequelize which is why I use the sync() method to create the tables on the first run. Unfortunately, when I run the application for the first time, I get an error that the table doesn't exist when using this code:
File: test.js
const database = require('./dbInit');
(async () => {
await database.testTable.max('id').then((maxId) => {
console.log(maxId);
});
})();
File: dbInit.js
const Sequelize = require('sequelize');
const sequelize = new Sequelize('mysql://root:root#localhost:3306/test');
const testTable = sequelize.import('testTable');
const database = {
sequelize: sequelize,
testTable: testTable,
};
sequelize
.authenticate()
.then(() => {
console.log('Connection to the database has been established successfully.');
})
.catch(error => {
console.error(error);
});
sequelize.sync();
module.exports = database;
File: testTable.js
const Sequelize = require('sequelize');
module.exports = (sequelize, DataTypes) => {
return sequelize.define('testTable',
{
id: {
type: Sequelize.BIGINT(19).UNSIGNED,
primaryKey: true,
autoIncrement: false,
}
}
);
};
When I run the code as is, without tables created, I can see from the logs that the query is run before the connection to the database is available:
> node .\test.js
Executing (default): SELECT 1+1 AS result
Executing (default): SELECT max(`id`) AS `max` FROM `testTables` AS `testTable`;
Connection to the database has been established successfully.
(node:1572) UnhandledPromiseRejectionWarning: SequelizeDatabaseError: Table 'test.testtables' doesn't exist
I have found a way to make it work by adding this like, just before the call to the DB (in test.js before the max('id') call):
await database.sequelize.sync();
Is there any other way to have the dbInit module completely independent and not having to add this sync() call inside all other files which will require database connectivity?
I've looked for sync module loading but it doesn't seem an option yet.

Because of async behavior all of ops that You want to do:
Connect
Sync
Do DB operations
You've to make it following way:
put model files to: db folder as: db/schemas/User.js
and make module file for db: db/index.js
const Sequelize = require('sequelize');
const sequelize = new Sequelize('mysql://root:root#localhost:3306/test');
const connect = async () => {
try {
await sequelize.authenticate();
await sequelize.sync();
console.log('Connection to the database has been established successfully.');
}
catch (error) {
console.error(error.message);
process.exit(-1);
}
});
const model = name => database.models[name];
const User = sequelize.import('./schemas/User');
const database = {
sequelize: sequelize,
models: {User},
connect,
model
};
module.exports = database;
and in test.js:
const db = require('./db');
(async () => {
await db.connect();
const User = db.model('User');
const id = await User.max('id');
console.log(id);
})();
P.S. forget about examples that used in web apps when developer does not care when db will connect and when express app will listen on port.
Your question is different - You want to do db query immediately, so You've to make sure connection and sync established successfully.

You can follow up my github repo Sequelize-DemoApp. It's a fully working full stack application made especially to demonstrate and understand Sequelize.js and it's integration with nodejs

Express.js: create additional mongodb DB in controller

I'm new to MongoDB.
When I create my node.js server I use only one DB connection (on start I connect to it).
But imagine: I have one database with some generic tables, and more databases - each for a custom client.
How can I create those DB at runtime?
start.js:
const mongoose = require("mongoose");
// import environmental variables from variables.env file
require("dotenv").config({ path: "variables.env" });
mongoose.connect(process.env.DATABASE);
mongoose.Promise = global.Promise;
mongoose.connection.on("error", err => {
console.error(`🚫 → ${err.message}`);
});
require("./models/MaintenanceType");
require("./models/Maintenance");
const app = require("./app");
app.set("port", process.env.PORT || 7777);
const server = app.listen(app.get("port"), () => {
console.log('started');
});
variables.env (example):
NODE_ENV=development
DATABASE=mongodb://db:qwe123#sometest.server.com:412345/webtest
PORT=1234
SECRET=webtest
KEY=webtestcom
and controller:
const mongoose = require("mongoose");
const Maintenance = mongoose.model("Maintenance");
exports.createMaintenance = async (req, res) => {
const maintenance = await new Maintenance(req.body).save();
// ALSO create a db and table if not exists for this client and use it somehow
res.json(maintenance);
};
is it possible to do?

You can create new connection
mongoose.connect('URI_FOR_ANOTHER_DATABASE')
But it's bad idea to create new connections, so the driver has a feature to use existing connections to query another database, for this purpose you can check useDb() method as shown here

Node.js Async/Await module export

I'm kinda new to module creation and was wondering about module.exports and waiting for async functions (like a mongo connect function for example) to complete and exporting the result. The variables get properly defined using async/await in the module, but when trying to log them by requiring the module, they show up as undefined. If someone could point me in the right direction, that'd be great. Here's the code I've got so far:
// module.js
const MongoClient = require('mongodb').MongoClient
const mongo_host = '127.0.0.1'
const mongo_db = 'test'
const mongo_port = '27017';
(async module => {
var client, db
var url = `mongodb://${mongo_host}:${mongo_port}/${mongo_db}`
try {
// Use connect method to connect to the Server
client = await MongoClient.connect(url, {
useNewUrlParser: true
})
db = client.db(mongo_db)
} catch (err) {
console.error(err)
} finally {
// Exporting mongo just to test things
console.log(client) // Just to test things I tried logging the client here and it works. It doesn't show 'undefined' like test.js does when trying to console.log it from there
module.exports = {
client,
db
}
}
})(module)
And here's the js that requires the module
// test.js
const {client} = require('./module')
console.log(client) // Logs 'undefined'
I'm fairly familiar with js and am still actively learning and looking into things like async/await and like features, but yeah... I can't really figure that one out

You have to export synchronously, so its impossible to export client and db directly. However you could export a Promise that resolves to client and db:
module.exports = (async function() {
const client = await MongoClient.connect(url, {
useNewUrlParser: true
});
const db = client.db(mongo_db);
return { client, db };
})();
So then you can import it as:
const {client, db} = await require("yourmodule");
(that has to be in an async function itself)
PS: console.error(err) is not a proper error handler, if you cant handle the error just crash

the solution provided above by #Jonas Wilms is working but requires to call requires in an async function each time we want to reuse the connection. an alternative way is to use a callback function to return the mongoDB client object.
mongo.js:
const MongoClient = require('mongodb').MongoClient;
const uri = "mongodb+srv://<user>:<pwd>#<host and port>?retryWrites=true";
const mongoClient = async function(cb) {
const client = await MongoClient.connect(uri, {
useNewUrlParser: true
});
cb(client);
};
module.exports = {mongoClient}
then we can use mongoClient method in a diffrent file(express route or any other js file).
app.js:
var client;
const mongo = require('path to mongo.js');
mongo.mongoClient((connection) => {
client = connection;
});
//declare express app and listen....
//simple post reuest to store a student..
app.post('/', async (req, res, next) => {
const newStudent = {
name: req.body.name,
description: req.body.description,
studentId: req.body.studetId,
image: req.body.image
};
try
{
await client.db('university').collection('students').insertOne({newStudent});
}
catch(err)
{
console.log(err);
return res.status(500).json({ error: err});
}
return res.status(201).json({ message: 'Student added'});
};