We Keep Coding

Bind problem in SQL query in Node, Express, Mysql2 app

I have been following a tutorial on setting up REST APIs in Node, using Express for an app that accesses an existing MariaDB database. My version only needs to read data and I have the DB co-located with the Node application (same host).
My goal for this entry-level example is to just access the data, using static SQL, so I can see it rendered in the web page by the JSON pritifier.
[Next, I want to present the data in a table (EJS?). Later, when I can get that to work, I'll add form controls (React?) to let a user specify start and end date bounds for the SQL query. Finally I'll aim to render the data as a line graph (D3js).]
The tutorial runs the web server successfully (it returns 'OK' on the base URL), but when I go to URL/solarData it tries an async function to getMultiple rows from the DB, it responds:
Bind parameters must not contain undefined. To pass SQL NULL specify JS null TypeError: Bind parameters must not contain undefined. To pass SQL NULL specify JS null
at /SunnyData/solarViz/node_modules/mysql2/lib/connection.js:628:17
at Array.forEach (<anonymous>)
at Connection.execute (/SunnyData/solarViz/node_modules/mysql2/lib/connection.js:620:22)
at /SunnyData/solarViz/node_modules/mysql2/promise.js:120:11
at new Promise (<anonymous>)
at PromiseConnection.execute (/SunnyData/solarViz/node_modules/mysql2/promise.js:117:12)
at Object.query (/SunnyData/solarViz/services/db.js:6:40)
at processTicksAndRejections (internal/process/task_queues.js:95:5)
at async Object.getMultiple (/SunnyData/solarViz/services/solarData.js:7:16)
at async /SunnyData/solarViz/routes/solarData.js:8:14
app.js:61
./app.js
const express = require('express');
const app = express();
const port = process.env.PORT || 3800;
const solarDataRouter = require('./routes/solarData');
app.use(express.json());
app.use(
express.urlencoded({
extended: true,
})
);
app.get('/', (req, res) => {
res.json({'message': 'ok'});
})
app.use('/solarData', solarDataRouter);
/* Error handler middleware */
app.use((err, req, res, next) => {
const statusCode = err.statusCode || 500;
console.error(err.message, err.stack);
res.status(statusCode).json({'message': err.message});
return;
});
app.listen(port, () => {
console.log(`Example app listening at http://localhost:${port}`)
});
./routes/solarData.js
const express = require('express');
const router = express.Router();
const solarData = require('../services/solarData');
/* GET solar data. */
router.get('/', async function(req, res, next) {
try {
res.json(await solarData.getMultiple(req.query.page));
} catch (err) {
console.error(`Error while getting solar data `, err.message);
next(err);
}
});
module.exports = router;
./config.js
const env = process.env;
const config = {
db: {
host: env.SUNNY_HOST,
user: env.SUNNY_USER,
password: env.SUNNY_PW,
database: env.SUNNY_DB,
},
listPerPage: env.LIST_PER_PAGE,
};
module.exports = config;
./services/solarData.js
const db = require('./db');
const helper = require('../helper');
const config = require('../config');
async function getMultiple(page = 1){
const offset = helper.getOffset(page, config.listPerPage);
const rows = await db.query(
`SELECT * FROM DTP LIMIT ?,?`, [offset, config.listPerPage]
);
const data = helper.emptyOrRows(rows);
const meta = {page};
return {
data,
meta
}
}
module.exports.getMultiple = getMultiple;
./services/db.js
const mysql = require('mysql2/promise');
const config = require('../config');
async function query(sql, params) {
const connection = await mysql.createConnection(config.db);
const [results, ] = await connection.execute(sql, params);
return results;
}
module.exports = {
query
}
I've left out the ./helper.js
Everything runs fine until I direct the webpage to /solarData. At that point I get the Debug Console (vscode) mentioned up-front
Searching seems to point at a mysql2 shortcoming/bug but not at a practical solution
If you respond, please describe the 'bind' mechanism, as I'm not sure what's going on.
Hope I've put enough info in. Please ask if I need to add anything else.

The error says
Bind parameters must not contain undefined.
It means that in the file ./services/solarData.js on the line
const rows = await db.query(
`SELECT * FROM DTP LIMIT ?,?`, [offset, config.listPerPage]
);
Some of the 2 variables is undefined, you need to check offset and config.listPerPage to be defined.
Just use
console.log('offset: ' + offset)
console.log('listPerPage: ' + config.listPerPage)
and you will find out what is undefined in your case

Mongo DB problem - connections accumulation

I have a problem with the approach I use to connect to Mondo DB.
I use the following method:
import { Db, MongoClient } from "mongodb";
let cachedConnection: { client: MongoClient; db: Db } | null = null;
export async function connectToDatabase(mongoUri?: string, database?: string) {
if (!mongoUri) {
throw new Error(
"Please define the MONGO_URI environment variable inside .env.local"
);
}
if (!database) {
throw new Error(
"Please define the DATABASE environment variable inside .env.local"
);
}
if (cachedConnection) return cachedConnection;
cachedConnection = await MongoClient.connect(mongoUri, {
useNewUrlParser: true,
useUnifiedTopology: true,
}).then((client) => ({
client,
db: client.db(database),
}));
return cachedConnection!;
}
Everytime I need to connect to MongoDB I do as follows:
const { db } = await connectToDatabase(config.URI, config.USERS_DATABASE);
const myUniversity = await db
.collection(config.MY_COLLECTION)
.findOne({})
Everything seems ok, so what is the problem?
The problem is that the connections to my DB don't close after I use them. In fact I thought that my server is stateless so after every time i use my DB, the connections end. But it is not true! They stay alive, and after few hours of using my app mongo atlas sends me an email saying that the limit is exceeded.
As you can see in this screenshot, this chart is ever growing. That means that connections stay on and they accumulate. How do you think I can solve this problem?
Keep in mind that it uses cachedConnection only if I use the same connection. If I call a different API from the first one it creates another connection and it doesn't enter in if (cachedConnection) block, but it goes forward till the end.

You can try this simple demo which will allow you to use the same connection throughout the application in different modules. There are three modules: the index.js is the starter program, the dbaccess.js is where you have code to create and maintain a connection which can be used again and again, and a apis.js module where you use the database connection to retrieve data.
index.js:
const express = require('express');
const mongo = require('./dbaccess');
const apis = require('./apis');
const app = express();
const init = async () => {
await mongo.connect();
app.listen(3000);
apis(app, mongo);
};
init();
dbaccess.js:
const { MongoClient } = require('mongodb');
class Mongo {
constructor() {
this.client = new MongoClient("mongodb://127.0.0.1:27017/", {
useNewUrlParser: true,
useUnifiedTopology: true
});
}
async connect() {
await this.client.connect();
console.log('Connected to MongoDB server.');
this.db = this.client.db('test');
console.log('Database:', this.db.databaseName);
}
}
module.exports = new Mongo();
apis.js:
module.exports = function(app, mongo) {
app.get('/', function(req, res) {
mongo.db.collection('users').find().limit(1).toArray(function(err, result) {
res.send('Doc: ' + JSON.stringify(result));
});
});
}
Change the appropriate values in the url, database name and collection name before trying.

can't obtain mongoose model inside one service from another using senecajs

I have two seneca services running on port 3000 and 3001. I am trying to create a separate mongoose connection from service on port 3000 create mongoose model and obtain it in service running on port 3001
connection-service.js
require('dotenv').config();
console.log("HOST",process.env.DB_PORT);
const seneca = require('seneca');
const plugins=require('./seneca_plugins/plugins');
const service=seneca();
service.use(plugins.user,{
collection:{UserAccountsCollection:"users"},
db:'accounts'
});
service.listen({port:3000,pin:'service:db_connection'});
plugin code for connection-service.js
const UserSchema = require('../db/models/user/user');
module.exports = async function (options) {
const dbConn = await require('./utils/dbConnections')(process.env.DB_HOST, process.env.DB_PORT, options.db);
this.add('service:db_connection,usage:GetUserAccountsConnection', async function (args, done) {
const UserModel = await dbConn.model('UserModel', UserSchema, options.collection.userAccountsCollection);
console.log(UserModel) //works
//done(null,{somekey:"someRandom_string_or_object"}) //works
done(null, {model:UserModel}); //passes empty object i.e. null,{}
})
}
service.js running as client
const service= require('seneca')();
const plugin=require('./seneca_plugins/plugin');
service.client({port:3000,pin:'service:db_connection'});
service.use(plugin);
plugin code for client service
module.exports = function (options) {
this.act('service:db_connection,usage:GetUserAccountsConnection', function (msg, reply) {
console.log("I am acting")
let model =reply.model //reply is returned as {}
console.log(model); //prints undefined in console
})
this.add(..... //other code follows
}
however when I use done(null,{somekey:"someRandom_string_or_object"}) it works but does not works when I pass model created done(null,{model:UserModel})

You can't mix callbacks and async/await. Try https://www.npmjs.com/package/seneca-promisify if you'd like to use async await.

Node.js Async/Await module export

I'm kinda new to module creation and was wondering about module.exports and waiting for async functions (like a mongo connect function for example) to complete and exporting the result. The variables get properly defined using async/await in the module, but when trying to log them by requiring the module, they show up as undefined. If someone could point me in the right direction, that'd be great. Here's the code I've got so far:
// module.js
const MongoClient = require('mongodb').MongoClient
const mongo_host = '127.0.0.1'
const mongo_db = 'test'
const mongo_port = '27017';
(async module => {
var client, db
var url = `mongodb://${mongo_host}:${mongo_port}/${mongo_db}`
try {
// Use connect method to connect to the Server
client = await MongoClient.connect(url, {
useNewUrlParser: true
})
db = client.db(mongo_db)
} catch (err) {
console.error(err)
} finally {
// Exporting mongo just to test things
console.log(client) // Just to test things I tried logging the client here and it works. It doesn't show 'undefined' like test.js does when trying to console.log it from there
module.exports = {
client,
db
}
}
})(module)
And here's the js that requires the module
// test.js
const {client} = require('./module')
console.log(client) // Logs 'undefined'
I'm fairly familiar with js and am still actively learning and looking into things like async/await and like features, but yeah... I can't really figure that one out

You have to export synchronously, so its impossible to export client and db directly. However you could export a Promise that resolves to client and db:
module.exports = (async function() {
const client = await MongoClient.connect(url, {
useNewUrlParser: true
});
const db = client.db(mongo_db);
return { client, db };
})();
So then you can import it as:
const {client, db} = await require("yourmodule");
(that has to be in an async function itself)
PS: console.error(err) is not a proper error handler, if you cant handle the error just crash

the solution provided above by #Jonas Wilms is working but requires to call requires in an async function each time we want to reuse the connection. an alternative way is to use a callback function to return the mongoDB client object.
mongo.js:
const MongoClient = require('mongodb').MongoClient;
const uri = "mongodb+srv://<user>:<pwd>#<host and port>?retryWrites=true";
const mongoClient = async function(cb) {
const client = await MongoClient.connect(uri, {
useNewUrlParser: true
});
cb(client);
};
module.exports = {mongoClient}
then we can use mongoClient method in a diffrent file(express route or any other js file).
app.js:
var client;
const mongo = require('path to mongo.js');
mongo.mongoClient((connection) => {
client = connection;
});
//declare express app and listen....
//simple post reuest to store a student..
app.post('/', async (req, res, next) => {
const newStudent = {
name: req.body.name,
description: req.body.description,
studentId: req.body.studetId,
image: req.body.image
};
try
{
await client.db('university').collection('students').insertOne({newStudent});
}
catch(err)
{
console.log(err);
return res.status(500).json({ error: err});
}
return res.status(201).json({ message: 'Student added'});
};

Mongodb + Node: When to close

I am working on a Koa + Mongodb backend. My question is: When should I close the db, or does Mongodb manage that because I am not closing any of them right now and it seems fine.
// app.js
const Koa = require('koa')
const database = require('./database')
const app = new Koa()
database
.connet()
.then(() => {app.listen(':8080')})
.catch((err) => {console.error(err)})
// ./database.js
const MongoClient = require('mongodb').MongoClient
const Model = require('./model')
class Database {
async connect() {
if (!db) {
db = await MongoClient.connect("localhost:27017")
this.item = new Model(db, 'item_collection')
}
}
}
module.exports = new Database()
// ./model.js
class Model {
constructor(db, collectionName) {
this.name = collectionName
this.database = database
}
async findAll() {
const result = await this.db.collection(this.name).find().toArray()
if (!result) {
throw new Error('error')
}
return result
}
}
module.exports = Model
I also ran a stress test using vegeta to make API request to the server at 100 request / second and the response time is good. So, am I worried about premature optimization here? If not, when should I close the db?

As Koa keeps running (and in your case listening on port 8080) you should not close the db connection.
If you are running scripts that are expected to end (tasks running on cron, etc) you should manually close the connection when you are finished with all of your db tasks.
You can take a look at this example for express.js (Koa's sister framework)