We Keep Coding

node-postgres library connection dies when it encounters an error

I have inherited a legacy system made in nodeJS and postgres. Whenever I encounter a database call error e.g let's say an insert violates a duplicate constraint the db client throws an error which I handle but its unable to make subsequent queries to the db and it hangs.
I have tried adding an error listener which recreates the client on error message but to no avail. I have many files calling the db so Its not ideal to recreate the client on each catch clause.
db connection class
const { Client } = require('pg');
const config = require('../../config');
const log = require('../../logger').LOG;
const client = new Client({
connectionString: config.dbUrl
});
client.on('error', err => {
log.info('client connection Error!'+ err.stack);
client = null;
client = new Client({
connectionString: config.dbUrl
});
client.connect();
});
client.on('end', () => {
log.info('client connection! End client sent');
});
client.on('notification', msg => {
log.info('client connection! notification message sent'+ msg);
});
client.connect();
// Export the Postgres Client module
module.exports = client;
sample query that is encountering
function createGame(gameHash) {
Model.query("INSERT INTO games (hash) values($1) RETURNING gId",[gameHash], function(err,db_res) {
if(err) {
log.info('Game record creation error: '+err.stack);
}
log.info('create record db resp: '+JSON.stringify(db_res));
gameId = db_res.rows[0].gId;
});
}
UPDATE:
so after reviemwing the logs keenly I have observed the issue occurs when the client decides to call an update command instead of the insert command in the query above. Interestingly the query passed to it is hard coded string clearly indicating insert but for some reason the update command is called.
2023-02-03 01:34:04 : create record db resp- with hash=>:
{"command":"UPDATE","rowCount":1,"oid":null,"rows":[],"fields":[],"_types":{"_types":{"arrayParser":{},
"builtins":{"BOOL":16,"BYTEA":17,"CHAR":18,"INT8":20,"INT2":21,"INT4":23,"REGPROC":24,"TEXT":25,"OID":26,"TID":27,"XID":28,"CID":29,"JSON":114,
"XML":142,"PG_NODE_TREE":194,"SMGR":210,"PATH":602,"POLYGON":604,
"CIDR":650,"FLOAT4":700,"FLOAT8":701,"ABSTIME":702,"RELTIME":703,
"TINTERVAL":704,"CIRCLE":718,"MACADDR8":774,"MONEY":790,"MACADDR":829,"INET":869,"ACLITEM":1033,"BPCHAR":1042,"VARCHAR":1043,"DATE":1082,
"TIME":1083,"TIMESTAMP":1114,"TIMESTAMPTZ":1184,"INTERVAL":1186,
"TIMETZ":1266,"BIT":1560,"VARBIT":1562,"NUMERIC":1700,"REFCURSOR":1790,"REGPROCEDURE":2202,"REGOPER":2203,"REGOPERATOR":2204,"REGCLASS":2205,"REGTYPE":2206,"UUID":2950,"TXID_SNAPSHOT":2970,"PG_LSN":3220,
"PG_NDISTINCT":3361,"PG_DEPENDENCIES":3402,"TSVECTOR":3614,"TSQUERY":3615,"GTSVECTOR":3642,"REGCONFIG":3734,"REGDICTIONARY":3769,
"JSONB":3802,"REGNAMESPACE":4089,"REGROLE":4096}},"text":{},"binary":{}},"RowCtor":null,"rowAsArray":false}

Node.Js MSSQL Query Timeout Expired

I am using Node Express API to run SQL queries to populate a dashboard of data. I am using the mssql-node package to do so. Sometimes it runs flawlessly, other times I get the following error:
[Error: [Microsoft][SQL Server Native Client 11.0]Query timeout expired]
I am creating a poolPromise with a connectionPool to the db, then I pass that object to my other controllers which run the specific queries to populate data. I run the server which initiates the db.js script and connects to MSSQL with a pool connection.
db.js:
// for connecting to sql server
const sql = require('mssql/msnodesqlv8');
// db config to connect via windows auth
const dbConfig = {
driver: 'msnodesqlv8',
connectionString: 'Driver={SQL Server Native Client 11.0};Server={my_server};Database={my_db};Trusted_Connection={yes};',
pool: {
idleTimeoutMillis: 60000
}
};
// create a connectionpool object to pass to controllers
// this should keep a sql connection open indefinitely that we can query when the server is running
const poolPromise = new sql.ConnectionPool(dbConfig)
.connect()
.then(pool => {
console.log('Connected to MSSQL');
return pool;
})
.catch(err => console.log('Database Connection Failed! Bad Config: ', err))
module.exports = { sql, poolPromise };
An example of one of my controllers and how I use the poolPromise object is below. I currently have about 7 of these controllers that run their own specific query to populate a specific element on the dashboard. The performance of the queries each run within 1-10 seconds (depending on current server load, as I am querying an enterprise production server/db, this can vary). As I mentioned earlier, the queries run flawlessly sometimes and I have no issues, but at other times I do have issues. Is this a symptom of me querying from a shared production server? Is it preferred to query from a server that has less load? Or am I doing something in my code that could be improved?
const { sql, poolPromise } = require('../db');
// function to get data
const getData = async (req, res) => {
try {
// create query parameters from user request
let id= req.query.id;
// create query from connectionPool
let pool = await poolPromise;
let qry = `
select * from tbl where id = #Id
`
let data = await pool.request()
.input('Id', sql.VarChar(sql.MAX), id)
.query(qry);
// send 200 status and return records
res.status(200);
res.send(data.recordset);
} catch(err) {
console.log('Error:');
console.log(err);
res.sendStatus(500);
}
};
module.exports = { getData };

Node JS mssql exporting database connection

I have hard time understanding why my code doesn't work. I am using node package mssql and want to have database pool connection initiation in separate file:
databaseConnection.js:
const sql = require("mssql/msnodesqlv8");
config = {
database: process.env.DB_NAME,
server: process.env.DB_SERVER,
driver: "msnodesqlv8",
options: {
trustedConnection: true
}
};
let pool = sql.connect(config);
module.exports = pool;
Then I have my express route file data.js
const express = require("express");
const router = express.Router();
const db = require("../configs/databaseConnection");
router.get("/dataList", async (req, res) => {
let allData = await db.request().query("select * from dataList");
console.log(allData);
res.render("dataList", { title: "Data list" });
});
module.exports = router;
However, when I start the server and go to the route I get error:
(node:13760) UnhandledPromiseRejectionWarning: TypeError: db.request is not a function
The thing is if I setup precisely as this example mssql documentation (where verything would be done in the route) it works. However, if database connection is in separate file it doesn't work.
I would appreciate any help understanding this
Regards,
Rokas

sql.connect returns a promise, so once we know that, we can either do a .then(result => ... or use await, for example:
If you want to store the db object at startup for later I'd suggest changing the line:
const db = require("../configs/databaseConnection");
to
let db = null;
require("../configs/databaseConnection").then(pool => {
db = pool;
});

Node.js Database Module with Sequelize

To make my code more readable, I'm trying to move all database related code into a single file. and use Sequelize as ORM. I would like that this file, when included provide a ready to use Database. Tables schemas are also managed by Sequelize which is why I use the sync() method to create the tables on the first run. Unfortunately, when I run the application for the first time, I get an error that the table doesn't exist when using this code:
File: test.js
const database = require('./dbInit');
(async () => {
await database.testTable.max('id').then((maxId) => {
console.log(maxId);
});
})();
File: dbInit.js
const Sequelize = require('sequelize');
const sequelize = new Sequelize('mysql://root:root#localhost:3306/test');
const testTable = sequelize.import('testTable');
const database = {
sequelize: sequelize,
testTable: testTable,
};
sequelize
.authenticate()
.then(() => {
console.log('Connection to the database has been established successfully.');
})
.catch(error => {
console.error(error);
});
sequelize.sync();
module.exports = database;
File: testTable.js
const Sequelize = require('sequelize');
module.exports = (sequelize, DataTypes) => {
return sequelize.define('testTable',
{
id: {
type: Sequelize.BIGINT(19).UNSIGNED,
primaryKey: true,
autoIncrement: false,
}
}
);
};
When I run the code as is, without tables created, I can see from the logs that the query is run before the connection to the database is available:
> node .\test.js
Executing (default): SELECT 1+1 AS result
Executing (default): SELECT max(`id`) AS `max` FROM `testTables` AS `testTable`;
Connection to the database has been established successfully.
(node:1572) UnhandledPromiseRejectionWarning: SequelizeDatabaseError: Table 'test.testtables' doesn't exist
I have found a way to make it work by adding this like, just before the call to the DB (in test.js before the max('id') call):
await database.sequelize.sync();
Is there any other way to have the dbInit module completely independent and not having to add this sync() call inside all other files which will require database connectivity?
I've looked for sync module loading but it doesn't seem an option yet.

Because of async behavior all of ops that You want to do:
Connect
Sync
Do DB operations
You've to make it following way:
put model files to: db folder as: db/schemas/User.js
and make module file for db: db/index.js
const Sequelize = require('sequelize');
const sequelize = new Sequelize('mysql://root:root#localhost:3306/test');
const connect = async () => {
try {
await sequelize.authenticate();
await sequelize.sync();
console.log('Connection to the database has been established successfully.');
}
catch (error) {
console.error(error.message);
process.exit(-1);
}
});
const model = name => database.models[name];
const User = sequelize.import('./schemas/User');
const database = {
sequelize: sequelize,
models: {User},
connect,
model
};
module.exports = database;
and in test.js:
const db = require('./db');
(async () => {
await db.connect();
const User = db.model('User');
const id = await User.max('id');
console.log(id);
})();
P.S. forget about examples that used in web apps when developer does not care when db will connect and when express app will listen on port.
Your question is different - You want to do db query immediately, so You've to make sure connection and sync established successfully.

You can follow up my github repo Sequelize-DemoApp. It's a fully working full stack application made especially to demonstrate and understand Sequelize.js and it's integration with nodejs

Issue connecting to my sql database using a REST api server with Node.js

I'm very new to coding servers and javascript in general but I'm currently trying to set up a REST api server and connect it to my sql database, for the moment I am doing everything locally. I am running ubuntu 18.04 while using NODE js. I have been able to successfully create a REST api and connect to it through an url of a webpage or with Postman. I have created a sql server database through my cmd terminal and have created test data on it. I've been looking at guides to connect the REST api to the database but I think the info I'm giving the api to connect is where my issue is occurring. I am starting with this below as my server.js where i have a folder Controller and a ProductController.js file where I'm handling the route /api/products .
var http = require('http');
var express = require('express');
var app = express();
var port = process.env.port || 3000;
var productController = require('./Controller/ProductController')();
app.use("/api/products", productController);
app.listen(port, function(){
var datetime = new Date();
var message = "Server running on Port:- " + port + " Started at :- " +
datetime;
console.log(message);
});
Below is my ProductController.js file. The issue might be here but I believe it is my next file called connect.js the table in my sql database is called 'data' hence the "SELECT * FROM data" part. when I try to GET this data in postman it displays the error i set up "Error while inserting data". so I believe when running I'm not getting data from sql so conn.close() is not being reached.
var express = require('express');
var router = express.Router();
var sql = require("mssql");
var conn = require("../connection/connect")();
var routes = function()
{
router.route('/')
.get(function(req, res)
{
conn.connect().then(function()
{
var sqlQuery = "SELECT * FROM data";
var req = new sql.Request(conn);
req.query(sqlQuery).then(function (recordset)
{
res.json(recordset.recordset);
conn.close();
})
.catch(function (err) {
conn.close();
res.status(400).send("Error while inserting data");
});
})
.catch(function (err) {
conn.close();
res.status(400).send("Error while inserting data");
});
});
return router;
};
module.exports = routes;
This is my connect.js file below. I have a password for root which is not *** but is correct on my machine. I have changed root's plug in to mysql_native_password in the mysql terminal. I think the server: part is wrong, I've tried commenting it out but still no connection. I do not have SQL Server Management Studio and have not found a way to get my sql server's name through the terminal. I've seen examples that seem to range of what info you need to give the api to connect. If someone has insight on that too that would be appreciated as well. My end goal is to eventually create GET and POST routes for the database and a function to manipulate the POST data but for now I'm just trying to get things connected so I can play around with the data being GET'ed. Thanks for any insight you can give, it is much appreciated.
var sql = require("mssql");
var connect = function()
{
var conn = new sql.ConnectionPool({
host: 'localhost'
user: 'root',
password: '********',
server: 'mysql',
database: 'test'
});
return conn;
};

Looks like you may have some errors in your connect.js file:
var conn = new sql.ConnectionPool({
host: 'localhost'
user: 'root',
password: '********',
server: 'mysql',
database: 'test'
});
should be in the format of:
const pool = new sql.ConnectionPool({
user: '...',
password: '...',
server: 'localhost',
database: '...'
})
Note that you currently have both host and server, looks like only server is needed. Also, server: 'mysql' doesn't make sense if you are connecting to a MSSQL database.
Source: node-mssql documentation

To diagnose the errors you should add some logging to your catch blocks:
.catch(function (err) {
console.log('connection error', err); //or Bunyan, Winston, Morgan, etc. logging library
conn.close();
let message = "Error while inserting data"
if (process.env.NODE_ENV === 'development') { //conditionally add error to result message
message += "\n"+err.toString());
}
res.status(500).send(message); //use 5xx for server problems, 4xx for things a user could potentially fix
});
And set NODE_ENV in your environment, for example in package.json:
"scripts": {
"start": "NODE_ENV=production node app.js"
"start-dev": "NODE_ENV=development node app.js"
}