We Keep Coding

Database Querying function not running asynchronously

So I created a function in my node server which takes in a query string, runs it on my db, and returns the results. I then wanted to use my function asynchronously using async await throughout my routes instead of having nested query within, nested query, within nested query etc.
So here is the code:
const runQuery = queryString => {
console.log("...runQuery")
db.query(queryString, (error, results, fields) => {
if (error) {
console.log("runQuery: FAILURE");
return error;
}
else {
console.log("runQuery: SUCCESS");
return(results);
}
})
}
register.post("/", async (req, res) => {
console.log(req.body);
const results = await runQuery("select * from test1");
res.send(results);
})
The database should have 3 entries, but unfortunately, it returns nothing. Meaning results is an empty variable when it is sent, meaning JS never properly waits for it to capture the db results. How can I use my function asynchronously, and how is this even feasible?

It seems your function "runQuery" does not return a promise, in fact, it's not returning anything. You are using "return" in the callback of the db.query function, not the function "runQuery" itself.
Since runQuery is performing an asynchronous operation, the result ought to be resolved via a promise (which is what the "await" in your request handler is looking for).
I'm not exactly sure but it seems you are using MySql, and I could not find anything on the npm page of the mysql package regarding the query being promisified, so we'll promisify it ourselves:
const runQuery = (queryString) => new Promise((resolve,reject) => {
console.log("...runQuery")
db.query(queryString, (error, results, fields) => {
if (error) {
console.error("runQuery: FAILURE");
reject(error);
} else {
console.log("runQuery: SUCCESS");
resolve(results);
}
})
})
register.post("/", async (req, res) => {
console.log(req.body);
try{
const results = await runQuery("select * from test1");
res.send(results);
}catch(e){
console.error(`ERROR THROWN BY runQuery:`,e);
res.status(500).send({
message: e.message || "INTERNAL SERVER ERROR"
})
}
})
Note that if an error occurs, our promisified function will reject the error and it will NOT be stored in the "results" variable in our request handler. Instead, an error will be thrown which needs to be handled. Therefore, it is always a good practice to put any async/await calls inside a try/catch.

Is creating a new promise with a async function call bad practice?

Snippets are from a node.js and mongoDB CRUD application.Github repo for full code. The code is working fine but unsure if my structure and use of promises and async await are bad practice.
handlers._newbies = {};
handlers._newbies.post = (parsedReq, res) => {
const newbie = JSON.parse(parsedReq.payload);
databaseCalls.create(newbie)
.then((result) => {
res.writeHead(200,{'Content-Type' : 'application/json'});
const resultToString = JSON.stringify(result.ops[0]);
res.write(resultToString);
res.end();
})
.catch(err => console.log(err));
};
const databaseCalls = {};
databaseCalls.create = (newbie) => {
return new Promise(async (resolve, reject) => {
try {
const client = await MongoClient.connect('mongodb://localhost:27017', { useNewUrlParser: true });
console.log("Connected correctly to server");
const db = client.db('Noob-List');
const result = await db.collection('newbies').insertOne(newbie);
client.close();
resolve(result);
} catch(err) {
console.log(err);
}
});
};
When the node server gets a POST request with the JSON payload, it calls the handlers._newbies.post handler which takes the payload and passed it to the
const newbie = JSON.parse(parsedReq.payload);
databaseCalls.create(newbie)
call. I want this database call to return a promise that holds the result of the db.collection('newbies').insertOne(newbie);
call. I was having trouble doing this with just returning the promise returned by the insertOne because after returning I cant call client.close();.
Again maybe what I have done here is fine but I haven't found anything online about creating promises with promises in them. Thank you for your time let me know what is unclear with my question.

It is considered an anti-pattern to be wrapping an existing promise in a manually created promise because there's just no reason to do so and it creates many an opportunities for error, particular in error handling.
And, in your case, you have several error handling issues.
If you get an error anywhere in your database code, you never resolve or reject the promise you are creating. This is a classic problem with the anti-pattern.
If you get an error after opening the DB, you don't close the DB
You don't communicate back an error to the caller.
Here's how you can do your .create() function without the anti-pattern and without the above problems:
databaseCalls.create = async function(newbie) {
let client;
try {
client = await MongoClient.connect('mongodb://localhost:27017', { useNewUrlParser: true });
console.log("Connected correctly to server");
const db = client.db('Noob-List');
return db.collection('newbies').insertOne(newbie);
} catch(err) {
// log error, but still reject the promise
console.log(err);
throw err;
} finally {
// clean up any open database
if (client) {
client.close();
}
}
}
Then, you would use this like:
databaseCalls.create(something).then(result => {
console.log("succeeded");'
}).catch(err => {
console.log(err);
});
FYI, I also modified some other things:
The database connection is closed, even in error conditions
The function returns a promise which is resolved with the result of .insertOne() (if there is a meaningful result there)
If there's an error, the returned promise is rejected with that error
Not particularly relevant to your issue with promises, but you will generally not want to open and close the DB connection on every operation. You can either use one lasting connection or create a pool of connections where you can fetch one from the pool and then put it back in the pool when done (most DBs have that type of feature for server-side work).

How to access data from function (node.js) [duplicate]

I’m new to node and mongo after 15 years of VB6 and MySql. I’m sure this is not what my final program will use but I need to get a basic understanding of how to call a function in another module and get results back.
I want a module to have a function to open a DB, find in a collection and return the results. I may want to add a couple more functions in that module for other collections too. For now I need it as simple as possible, I can add error handlers, etc later. I been on this for days trying different methods, module.exports={… around the function and with out it, .send, return all with no luck. I understand it’s async so the program may have passed the display point before the data is there.
Here’s what I’ve tried with Mongo running a database of db1 with a collection of col1.
Db1.js
var MongoClient = require('mongodb').MongoClient;
module.exports = {
FindinCol1 : function funk1(req, res) {
MongoClient.connect("mongodb://localhost:27017/db1", function (err,db) {
if (err) {
return console.dir(err);
}
var collection = db.collection('col1');
collection.find().toArray(function (err, items) {
console.log(items);
// res.send(items);
}
);
});
}
};
app.js
a=require('./db1');
b=a.FindinCol1();
console.log(b);
Console.log(items) works when the 'FindinCol1' calls but not console.log(b)(returns 'undefined') so I'm not getting the return or I'm pasted it by the time is returns. I’ve read dozens of post and watched dozens of videos but I'm still stuck at this point. Any help would be greatly appreciated.

As mentioned in another answer, this code is asynchronous, you can't simply return the value you want down the chain of callbacks (nested functions). You need to expose some interface that lets you signal the calling code once you have the value desired (hence, calling them back, or callback).
There is a callback example provided in another answer, but there is an alternative option definitely worth exploring: promises.
Instead of a callback function you call with the desired results, the module returns a promise that can enter two states, fulfilled or rejected. The calling code waits for the promise to enter one of these two states, the appropriate function being called when it does. The module triggers the state change by resolveing or rejecting. Anyways, here is an example using promises:
Db1.js:
// db1.js
var MongoClient = require('mongodb').MongoClient;
/*
node.js has native support for promises in recent versions.
If you are using an older version there are several libraries available:
bluebird, rsvp, Q. I'll use rsvp here as I'm familiar with it.
*/
var Promise = require('rsvp').Promise;
module.exports = {
FindinCol1: function() {
return new Promise(function(resolve, reject) {
MongoClient.connect('mongodb://localhost:27017/db1', function(err, db) {
if (err) {
reject(err);
} else {
resolve(db);
}
}
}).then(function(db) {
return new Promise(function(resolve, reject) {
var collection = db.collection('col1');
collection.find().toArray(function(err, items) {
if (err) {
reject(err);
} else {
console.log(items);
resolve(items);
}
});
});
});
}
};
// app.js
var db = require('./db1');
db.FindinCol1().then(function(items) {
console.info('The promise was fulfilled with items!', items);
}, function(err) {
console.error('The promise was rejected', err, err.stack);
});
Now, more up to date versions of the node.js mongodb driver have native support for promises, you don't have to do any work to wrap callbacks in promises like above. This is a much better example if you are using an up to date driver:
// db1.js
var MongoClient = require('mongodb').MongoClient;
module.exports = {
FindinCol1: function() {
return MongoClient.connect('mongodb://localhost:27017/db1').then(function(db) {
var collection = db.collection('col1');
return collection.find().toArray();
}).then(function(items) {
console.log(items);
return items;
});
}
};
// app.js
var db = require('./db1');
db.FindinCol1().then(function(items) {
console.info('The promise was fulfilled with items!', items);
}, function(err) {
console.error('The promise was rejected', err, err.stack);
});
Promises provide an excellent method for asynchronous control flow, I highly recommend spending some time familiarizing yourself with them.

Yes, this is an async code and with a return you will get the MongoClient object or nothing, based on where you put.
You should use a callback parameter:
module.exports = {
FindinCol1 : function funk1(callback) {
MongoClient.connect("mongodb://localhost:27017/db1", function (err,db) {
if (err) {
return console.dir(err);
}
var collection = db.collection('col1');
collection.find().toArray(function (err, items) {
console.log(items);
return callback(items);
});
});
}
};
Pass a callback function to FindinCol1:
a.FindinCol1(function(items) {
console.log(items);
});
I suggest you to check this article:
https://docs.nodejitsu.com/articles/getting-started/control-flow/what-are-callbacks

Storing data from MongoDB in Javascript Objects in Node.js [duplicate]

I’m new to node and mongo after 15 years of VB6 and MySql. I’m sure this is not what my final program will use but I need to get a basic understanding of how to call a function in another module and get results back.
I want a module to have a function to open a DB, find in a collection and return the results. I may want to add a couple more functions in that module for other collections too. For now I need it as simple as possible, I can add error handlers, etc later. I been on this for days trying different methods, module.exports={… around the function and with out it, .send, return all with no luck. I understand it’s async so the program may have passed the display point before the data is there.
Here’s what I’ve tried with Mongo running a database of db1 with a collection of col1.
Db1.js
var MongoClient = require('mongodb').MongoClient;
module.exports = {
FindinCol1 : function funk1(req, res) {
MongoClient.connect("mongodb://localhost:27017/db1", function (err,db) {
if (err) {
return console.dir(err);
}
var collection = db.collection('col1');
collection.find().toArray(function (err, items) {
console.log(items);
// res.send(items);
}
);
});
}
};
app.js
a=require('./db1');
b=a.FindinCol1();
console.log(b);
Console.log(items) works when the 'FindinCol1' calls but not console.log(b)(returns 'undefined') so I'm not getting the return or I'm pasted it by the time is returns. I’ve read dozens of post and watched dozens of videos but I'm still stuck at this point. Any help would be greatly appreciated.

As mentioned in another answer, this code is asynchronous, you can't simply return the value you want down the chain of callbacks (nested functions). You need to expose some interface that lets you signal the calling code once you have the value desired (hence, calling them back, or callback).
There is a callback example provided in another answer, but there is an alternative option definitely worth exploring: promises.
Instead of a callback function you call with the desired results, the module returns a promise that can enter two states, fulfilled or rejected. The calling code waits for the promise to enter one of these two states, the appropriate function being called when it does. The module triggers the state change by resolveing or rejecting. Anyways, here is an example using promises:
Db1.js:
// db1.js
var MongoClient = require('mongodb').MongoClient;
/*
node.js has native support for promises in recent versions.
If you are using an older version there are several libraries available:
bluebird, rsvp, Q. I'll use rsvp here as I'm familiar with it.
*/
var Promise = require('rsvp').Promise;
module.exports = {
FindinCol1: function() {
return new Promise(function(resolve, reject) {
MongoClient.connect('mongodb://localhost:27017/db1', function(err, db) {
if (err) {
reject(err);
} else {
resolve(db);
}
}
}).then(function(db) {
return new Promise(function(resolve, reject) {
var collection = db.collection('col1');
collection.find().toArray(function(err, items) {
if (err) {
reject(err);
} else {
console.log(items);
resolve(items);
}
});
});
});
}
};
// app.js
var db = require('./db1');
db.FindinCol1().then(function(items) {
console.info('The promise was fulfilled with items!', items);
}, function(err) {
console.error('The promise was rejected', err, err.stack);
});
Now, more up to date versions of the node.js mongodb driver have native support for promises, you don't have to do any work to wrap callbacks in promises like above. This is a much better example if you are using an up to date driver:
// db1.js
var MongoClient = require('mongodb').MongoClient;
module.exports = {
FindinCol1: function() {
return MongoClient.connect('mongodb://localhost:27017/db1').then(function(db) {
var collection = db.collection('col1');
return collection.find().toArray();
}).then(function(items) {
console.log(items);
return items;
});
}
};
// app.js
var db = require('./db1');
db.FindinCol1().then(function(items) {
console.info('The promise was fulfilled with items!', items);
}, function(err) {
console.error('The promise was rejected', err, err.stack);
});
Promises provide an excellent method for asynchronous control flow, I highly recommend spending some time familiarizing yourself with them.

Yes, this is an async code and with a return you will get the MongoClient object or nothing, based on where you put.
You should use a callback parameter:
module.exports = {
FindinCol1 : function funk1(callback) {
MongoClient.connect("mongodb://localhost:27017/db1", function (err,db) {
if (err) {
return console.dir(err);
}
var collection = db.collection('col1');
collection.find().toArray(function (err, items) {
console.log(items);
return callback(items);
});
});
}
};
Pass a callback function to FindinCol1:
a.FindinCol1(function(items) {
console.log(items);
});
I suggest you to check this article:
https://docs.nodejitsu.com/articles/getting-started/control-flow/what-are-callbacks

Replacing callbacks with promises in Node.js

I have a simple node module which connects to a database and has several functions to receive data, for example this function:
dbConnection.js:
import mysql from 'mysql';
const connection = mysql.createConnection({
host: 'localhost',
user: 'user',
password: 'password',
database: 'db'
});
export default {
getUsers(callback) {
connection.connect(() => {
connection.query('SELECT * FROM Users', (err, result) => {
if (!err){
callback(result);
}
});
});
}
};
The module would be called this way from a different node module:
app.js:
import dbCon from './dbConnection.js';
dbCon.getUsers(console.log);
I would like to use promises instead of callbacks in order to return the data.
So far I've read about nested promises in the following thread: Writing Clean Code With Nested Promises, but I couldn't find any solution that is simple enough for this use case.
What would be the correct way to return result using a promise?

Using the Promise class
I recommend to take a look at MDN's Promise docs which offer a good starting point for using Promises. Alternatively, I am sure there are many tutorials available online.:)
Note: Modern browsers already support ECMAScript 6 specification of Promises (see the MDN docs linked above) and I assume that you want to use the native implementation, without 3rd party libraries.
As for an actual example...
The basic principle works like this:
Your API is called
You create a new Promise object, this object takes a single function as constructor parameter
Your provided function is called by the underlying implementation and the function is given two functions - resolve and reject
Once you do your logic, you call one of these to either fullfill the Promise or reject it with an error
This might seem like a lot so here is an actual example.
exports.getUsers = function getUsers () {
// Return the Promise right away, unless you really need to
// do something before you create a new Promise, but usually
// this can go into the function below
return new Promise((resolve, reject) => {
// reject and resolve are functions provided by the Promise
// implementation. Call only one of them.
// Do your logic here - you can do WTF you want.:)
connection.query('SELECT * FROM Users', (err, result) => {
// PS. Fail fast! Handle errors first, then move to the
// important stuff (that's a good practice at least)
if (err) {
// Reject the Promise with an error
return reject(err)
}
// Resolve (or fulfill) the promise with data
return resolve(result)
})
})
}
// Usage:
exports.getUsers() // Returns a Promise!
.then(users => {
// Do stuff with users
})
.catch(err => {
// handle errors
})
Using the async/await language feature (Node.js >=7.6)
In Node.js 7.6, the v8 JavaScript compiler was upgraded with async/await support. You can now declare functions as being async, which means they automatically return a Promise which is resolved when the async function completes execution. Inside this function, you can use the await keyword to wait until another Promise resolves.
Here is an example:
exports.getUsers = async function getUsers() {
// We are in an async function - this will return Promise
// no matter what.
// We can interact with other functions which return a
// Promise very easily:
const result = await connection.query('select * from users')
// Interacting with callback-based APIs is a bit more
// complicated but still very easy:
const result2 = await new Promise((resolve, reject) => {
connection.query('select * from users', (err, res) => {
return void err ? reject(err) : resolve(res)
})
})
// Returning a value will cause the promise to be resolved
// with that value
return result
}

With bluebird you can use Promise.promisifyAll (and Promise.promisify) to add Promise ready methods to any object.
var Promise = require('bluebird');
// Somewhere around here, the following line is called
Promise.promisifyAll(connection);
exports.getUsersAsync = function () {
return connection.connectAsync()
.then(function () {
return connection.queryAsync('SELECT * FROM Users')
});
};
And use like this:
getUsersAsync().then(console.log);
or
// Spread because MySQL queries actually return two resulting arguments,
// which Bluebird resolves as an array.
getUsersAsync().spread(function(rows, fields) {
// Do whatever you want with either rows or fields.
});
Adding disposers
Bluebird supports a lot of features, one of them is disposers, it allows you to safely dispose of a connection after it ended with the help of Promise.using and Promise.prototype.disposer. Here's an example from my app:
function getConnection(host, user, password, port) {
// connection was already promisified at this point
// The object literal syntax is ES6, it's the equivalent of
// {host: host, user: user, ... }
var connection = mysql.createConnection({host, user, password, port});
return connection.connectAsync()
// connect callback doesn't have arguments. return connection.
.return(connection)
.disposer(function(connection, promise) {
//Disposer is used when Promise.using is finished.
connection.end();
});
}
Then use it like this:
exports.getUsersAsync = function () {
return Promise.using(getConnection()).then(function (connection) {
return connection.queryAsync('SELECT * FROM Users')
});
};
This will automatically end the connection once the promise resolves with the value (or rejects with an Error).

Node.js version 8.0.0+:
You don't have to use bluebird to promisify the node API methods anymore. Because, from version 8+ you can use native util.promisify:
const util = require('util');
const connectAsync = util.promisify(connection.connectAsync);
const queryAsync = util.promisify(connection.queryAsync);
exports.getUsersAsync = function () {
return connectAsync()
.then(function () {
return queryAsync('SELECT * FROM Users')
});
};
Now, don't have to use any 3rd party lib to do the promisify.

Assuming your database adapter API doesn't output Promises itself you can do something like:
exports.getUsers = function () {
var promise;
promise = new Promise();
connection.connect(function () {
connection.query('SELECT * FROM Users', function (err, result) {
if(!err){
promise.resolve(result);
} else {
promise.reject(err);
}
});
});
return promise.promise();
};
If the database API does support Promises you could do something like: (here you see the power of Promises, your callback fluff pretty much disappears)
exports.getUsers = function () {
return connection.connect().then(function () {
return connection.query('SELECT * FROM Users');
});
};
Using .then() to return a new (nested) promise.
Call with:
module.getUsers().done(function (result) { /* your code here */ });
I used a mockup API for my Promises, your API might be different. If you show me your API I can tailor it.

2019:
Use that native module const {promisify} = require('util'); to conver plain old callback pattern to promise pattern so you can get benfit from async/await code
const {promisify} = require('util');
const glob = promisify(require('glob'));
app.get('/', async function (req, res) {
const files = await glob('src/**/*-spec.js');
res.render('mocha-template-test', {files});
});

When setting up a promise you take two parameters, resolve and reject. In the case of success, call resolve with the result, in the case of failure call reject with the error.
Then you can write:
getUsers().then(callback)
callback will be called with the result of the promise returned from getUsers, i.e. result

Using the Q library for example:
function getUsers(param){
var d = Q.defer();
connection.connect(function () {
connection.query('SELECT * FROM Users', function (err, result) {
if(!err){
d.resolve(result);
}
});
});
return d.promise;
}

Below code works only for node -v > 8.x
I use this Promisified MySQL middleware for Node.js
read this article Create a MySQL Database Middleware with Node.js 8 and Async/Await
database.js
var mysql = require('mysql');
// node -v must > 8.x
var util = require('util');
// !!!!! for node version < 8.x only !!!!!
// npm install util.promisify
//require('util.promisify').shim();
// -v < 8.x has problem with async await so upgrade -v to v9.6.1 for this to work.
// connection pool https://github.com/mysqljs/mysql [1]
var pool = mysql.createPool({
connectionLimit : process.env.mysql_connection_pool_Limit, // default:10
host : process.env.mysql_host,
user : process.env.mysql_user,
password : process.env.mysql_password,
database : process.env.mysql_database
})
// Ping database to check for common exception errors.
pool.getConnection((err, connection) => {
if (err) {
if (err.code === 'PROTOCOL_CONNECTION_LOST') {
console.error('Database connection was closed.')
}
if (err.code === 'ER_CON_COUNT_ERROR') {
console.error('Database has too many connections.')
}
if (err.code === 'ECONNREFUSED') {
console.error('Database connection was refused.')
}
}
if (connection) connection.release()
return
})
// Promisify for Node.js async/await.
pool.query = util.promisify(pool.query)
module.exports = pool
You must upgrade node -v > 8.x
you must use async function to be able to use await.
example:
var pool = require('./database')
// node -v must > 8.x, --> async / await
router.get('/:template', async function(req, res, next)
{
...
try {
var _sql_rest_url = 'SELECT * FROM arcgis_viewer.rest_url WHERE id='+ _url_id;
var rows = await pool.query(_sql_rest_url)
_url = rows[0].rest_url // first record, property name is 'rest_url'
if (_center_lat == null) {_center_lat = rows[0].center_lat }
if (_center_long == null) {_center_long= rows[0].center_long }
if (_center_zoom == null) {_center_zoom= rows[0].center_zoom }
_place = rows[0].place
} catch(err) {
throw new Error(err)
}