We Keep Coding

Mongoose pass data out of withTransaction helper

Introduction
Hey there,
I am trying to pass out data from the mongoose withTransaction callback. Right now, I am using the following code which implements callbacks:
const transactionSession = await mongoose.startSession()
await transactionSession.withTransaction(async (tSession) => {
try {
// MARK Transaction writes & reads removed for brevity
console.log("Successfully performed transaction!")
cb(null, "Any test data")
return Promise.resolve()
} catch (error) {
console.log("Transaction aborted due to error:", error)
cb(error)
return Promise.reject()
}
})
} catch (error) {
console.log(error)
return cb(error)
}
A more detailed snippet of the withTransaction helper in use can be found here.
A link to the official Mongoose documentation regarding the withTransaction helper can be found here.
At the moment, I am using a callback to pass out data from the withTransactioncallback:
cb(null, "Any test data")
However, the problem is that naturally the callback is executed first, before the Promise.resolve() is returned. This means, that (in my case) a success response is sent back to the client before any necessary database writes are committed:
// this is executed first - the callback will send back a response to the client
cb(null, "Any test data")
// only now, after the response already got sent to the client, the transaction is committed.
return Promise.resolve()
Why I think this is a problem:
Honestly, I am not sure. It just doesn't feel right to send back a success-response to the client, if there hasn't been any database write at that time. Does anybody know the appropriate way to deal with this specific use-case?
I thought about passing data out of the withTransaction helper using something like this:
const transactionResult = await transactionSession.withTransaction({...})
I've tried it, and the response is a CommandResult of MongoDB, which does not include any of the data I included in the resolved promise.
Summary
Is it a problem, if a success response is sent back to the client before the transaction is committed? If so, what is the appropriate way to pass out data from the withTransaction helper and thereby committing the transaction before sending back a response?
I would be thankful for any advice I get.

It looks like there is some confusion here as to how to correctly use Promises, on several levels.
Callback and Promise are being used incorrectly
If the function is supposed to accept a callback, don't return a Promise. If the function is supposed to return a Promise, use the callback given by the Promise:
const transactionSession = await mongoose.startSession()
await transactionSession.withTransaction( (tSession) => {
return new Promise( (resolve, reject) => {
//using Node-style callback
doSomethingAsync( (err, testData) => {
if(err) {
reject(err);
} else {
resolve(testData); //this is the equivalent of cb(null, "Any test data")
}
});
})
Let's look at this in more detail:
return new Promise( (resolve, reject) => { This creates a new Promise, and the Promise is giving you two callbacks to use. resolve is a callback to indicate success. You pass it the object you'd like to return. Note that I've removed the async keyword (more on this later).
For example:
const a = new Promise( (resolve, reject) => resolve(5) );
a.then( (result) => result == 5 ); //true
(err, testData) => { This function is used to map the Node-style cb(err, result) to the Promise's callbacks.
Try/catch are being used incorrectly.
Try/catch can only be used for synchronous statements. Let's compare a synchronous call, a Node-style (i.e. cb(err, result)) asynchronous callback, a Promise, and using await:
Synchronous:
try {
let a = doSomethingSync();
} catch(err) {
handle(err);
}
Async:
doSomethingAsync( (err, result) => {
if (err) {
handle(err);
} else {
let a = result;
}
});
Promise:
doSomethingPromisified()
.then( (result) => {
let a = result;
})
.catch( (err) => {
handle(err);
});
Await. Await can be used with any function that returns a Promise, and lets you handle the code as if it were synchronous:
try {
let a = await doSomethingPromisified();
} catch(err) {
handle(err);
}
Additional Info
Promise.resolve()
Promise.resolve() creates a new Promise and resolves that Promise with an undefined value. This is shorthand for:
new Promise( (resolve, reject) => resolve(undefined) );
The callback equivalent of this would be:
cb(err, undefined);
async
async goes with await. If you are using await in a function, that function must be declared to be async.
Just as await unwraps a Promise (resolve into a value, and reject into an exception), async wraps code into a Promise. A return value statement gets translated into Promise.resolve(value), and a thrown exception throw e gets translated into Promise.reject(e).
Consider the following code
async () => {
return doSomethingSync();
}
The code above is equivalent to this:
() => {
const p = new Promise(resolve, reject);
try {
const value = doSomethingSync();
p.resolve(value);
} catch(e) {
p.reject(e);
}
return p;
}
If you call either of the above functions without await, you will get back a Promise. If you await either of them, you will be returned a value, or an exception will be thrown.

Database Querying function not running asynchronously

So I created a function in my node server which takes in a query string, runs it on my db, and returns the results. I then wanted to use my function asynchronously using async await throughout my routes instead of having nested query within, nested query, within nested query etc.
So here is the code:
const runQuery = queryString => {
console.log("...runQuery")
db.query(queryString, (error, results, fields) => {
if (error) {
console.log("runQuery: FAILURE");
return error;
}
else {
console.log("runQuery: SUCCESS");
return(results);
}
})
}
register.post("/", async (req, res) => {
console.log(req.body);
const results = await runQuery("select * from test1");
res.send(results);
})
The database should have 3 entries, but unfortunately, it returns nothing. Meaning results is an empty variable when it is sent, meaning JS never properly waits for it to capture the db results. How can I use my function asynchronously, and how is this even feasible?

It seems your function "runQuery" does not return a promise, in fact, it's not returning anything. You are using "return" in the callback of the db.query function, not the function "runQuery" itself.
Since runQuery is performing an asynchronous operation, the result ought to be resolved via a promise (which is what the "await" in your request handler is looking for).
I'm not exactly sure but it seems you are using MySql, and I could not find anything on the npm page of the mysql package regarding the query being promisified, so we'll promisify it ourselves:
const runQuery = (queryString) => new Promise((resolve,reject) => {
console.log("...runQuery")
db.query(queryString, (error, results, fields) => {
if (error) {
console.error("runQuery: FAILURE");
reject(error);
} else {
console.log("runQuery: SUCCESS");
resolve(results);
}
})
})
register.post("/", async (req, res) => {
console.log(req.body);
try{
const results = await runQuery("select * from test1");
res.send(results);
}catch(e){
console.error(`ERROR THROWN BY runQuery:`,e);
res.status(500).send({
message: e.message || "INTERNAL SERVER ERROR"
})
}
})
Note that if an error occurs, our promisified function will reject the error and it will NOT be stored in the "results" variable in our request handler. Instead, an error will be thrown which needs to be handled. Therefore, it is always a good practice to put any async/await calls inside a try/catch.

Is creating a new promise with a async function call bad practice?

Snippets are from a node.js and mongoDB CRUD application.Github repo for full code. The code is working fine but unsure if my structure and use of promises and async await are bad practice.
handlers._newbies = {};
handlers._newbies.post = (parsedReq, res) => {
const newbie = JSON.parse(parsedReq.payload);
databaseCalls.create(newbie)
.then((result) => {
res.writeHead(200,{'Content-Type' : 'application/json'});
const resultToString = JSON.stringify(result.ops[0]);
res.write(resultToString);
res.end();
})
.catch(err => console.log(err));
};
const databaseCalls = {};
databaseCalls.create = (newbie) => {
return new Promise(async (resolve, reject) => {
try {
const client = await MongoClient.connect('mongodb://localhost:27017', { useNewUrlParser: true });
console.log("Connected correctly to server");
const db = client.db('Noob-List');
const result = await db.collection('newbies').insertOne(newbie);
client.close();
resolve(result);
} catch(err) {
console.log(err);
}
});
};
When the node server gets a POST request with the JSON payload, it calls the handlers._newbies.post handler which takes the payload and passed it to the
const newbie = JSON.parse(parsedReq.payload);
databaseCalls.create(newbie)
call. I want this database call to return a promise that holds the result of the db.collection('newbies').insertOne(newbie);
call. I was having trouble doing this with just returning the promise returned by the insertOne because after returning I cant call client.close();.
Again maybe what I have done here is fine but I haven't found anything online about creating promises with promises in them. Thank you for your time let me know what is unclear with my question.

It is considered an anti-pattern to be wrapping an existing promise in a manually created promise because there's just no reason to do so and it creates many an opportunities for error, particular in error handling.
And, in your case, you have several error handling issues.
If you get an error anywhere in your database code, you never resolve or reject the promise you are creating. This is a classic problem with the anti-pattern.
If you get an error after opening the DB, you don't close the DB
You don't communicate back an error to the caller.
Here's how you can do your .create() function without the anti-pattern and without the above problems:
databaseCalls.create = async function(newbie) {
let client;
try {
client = await MongoClient.connect('mongodb://localhost:27017', { useNewUrlParser: true });
console.log("Connected correctly to server");
const db = client.db('Noob-List');
return db.collection('newbies').insertOne(newbie);
} catch(err) {
// log error, but still reject the promise
console.log(err);
throw err;
} finally {
// clean up any open database
if (client) {
client.close();
}
}
}
Then, you would use this like:
databaseCalls.create(something).then(result => {
console.log("succeeded");'
}).catch(err => {
console.log(err);
});
FYI, I also modified some other things:
The database connection is closed, even in error conditions
The function returns a promise which is resolved with the result of .insertOne() (if there is a meaningful result there)
If there's an error, the returned promise is rejected with that error
Not particularly relevant to your issue with promises, but you will generally not want to open and close the DB connection on every operation. You can either use one lasting connection or create a pool of connections where you can fetch one from the pool and then put it back in the pool when done (most DBs have that type of feature for server-side work).

How to convert callback code to async/await? [duplicate]

This question already has answers here:
Nodejs Mongoose - how to avoid callback hell?
(3 answers)
Closed 5 years ago.
I am working with Mongoose ODM for MongoDB in a node express-based web application which provides its API in a callback fashion. This fashion creates a callback hell to populate documents.
Edit 1: I have added the current working code.
User.remove({}, (err) => {
if (err) return console.error(err)
let admin = new User({
email: 'admin#dap.com',
password: 'dapdap'
})
admin.save((err, admin) => {
if (err) return console.error(err)
console.info('Success')
mongoose.disconnect()
})
})
Is there an elegant way to convert the callbacks into async/await fashion in javascript in general? And in mongoose environment?

Would you be able to configure mongoose to use bluebird?
mongoose.Promise = require('bluebird');
Then you could get fancy and do:
User.remove().then(() => {
let admin = new User({ ... });
admin.save().then((admin) => {
}).catch((err) => {
//Saving admin error handling
});
}).catch((err) {
//User remove error handling
});

The same code you wrote above can be written as a promise. From the original docs,
"Mongoose queries are not promises. However, they do have a .then() function for yield and async/await. If you need a fully-fledged promise, use the .exec() function."
Thus you can use the async await pattern as follows,
async function removeUser () {
// Surround the following call with a try catch
await User.remove({});
}
EDIT:
One answer here says about plugging in blue bird promises. While this is a good approach, it is optional. You can do the same without plugging in your own Promise library.

You can use a Promise:
const userRemovePromise = new Promise((resolve, reject) => {
User.remove({}, (err) => {
if (err) reject();
let admin = new User({
email: 'admin#dap.com',
password: 'dapdap'
})
admin.save((err, admin) => {
if (err) reject();
resolve();
})
});
});
userRemovePromise.then(function(){
mongoose.disconnect();
});

As mongoose's default promise library mpromise is now deprecated so you can plugin your own promise library to make mongoose's operations work with await/async.
Mongoose queries are not promises. However, they do have a .then() function for yield and async/await. If you need a fully-fledged promise, use the .exec() function.
mongoose.Promise = require('bluebird'); bluebird promise
// mongoose.Promise = global.Promise; // default js promise - 4x- time slower than bluebird
async function removeUser () {
try {
await User.remove({}).exec();
let admin = new User({
email: 'admin#dap.com',
password: 'dapdap'
})
await admin.save();
} catch (e) {
console.log(e);
}
}
Official Doc: mongoosejs.com/docs/promises.html

Recent versions of Mongoose returns a Promise as well as providing the regular callback-style pattern. Since async functions are syntatic sugar over Promises, you can await calls to Mongoose methods.
async function clearUsers () {
try {
await User.remove({})
let admin = new User({
email: 'admin#dap.com',
password: 'dapdap'
})
await admin.save()
console.info('Success')
mongoose.disconnect()
} catch (e) {
console.error(e)
}
}
Some things to keep in mind:
async functions always returns a Promise. If you don't return anything from an async function, it will still return a Promise that resolves when execution of that function finishes, but will not resolve with any value.
try/catch in an async function works the same as for regular, synchronous code. If any function call in the try body throw or return a Promise that rejects, execution stops right on that line and proceeds to the catch body.
Rejected promises "trickle" up the function call chain. This means that the top-most callee can handle errors. See the below example:
This is an anti-pattern that should be avoided unless you absolutely need to handle an error in a specific function, possibly to provide a return value from a different source:
async function fn1 () {
throw new Error("Something went wrong")
}
async function fn2 () {
try {
await fn1()
} catch (e) {
throw e
}
}
async function fn3 () {
try {
await fn2()
} catch (e) {
throw e
}
}
async function run () {
try {
await fn3()
} catch (e) {
console.error(e)
}
}
The above could rather be implemented like the below and still catch the error, not resulting in a runtime panic/crash:
async function fn1 () {
throw new Error("Something went wrong")
}
function fn2 () {
return fn1()
}
function fn3 () {
return fn2()
}
async function run () {
try {
await fn3()
} catch (e) {
console.error(e)
}
}
There are multiple ways to write the above code that will all be valid, so I recommend that you explore these.
Keeping the above examples in mind, your function clearUsers() could then be rewritten to this:
async function clearUsers () {
await User.remove({})
let admin = new User({
email: 'admin#dap.com',
password: 'dapdap'
})
await admin.save()
mongoose.disconnect()
}
And then possibly called in two different ways;
By interacting with the Promise returned directly:
clearUsers()
.then(() => {
console.log('Success')
})
.catch((e) => {
console.error(e)
})
Or from within another async function:
(async function () {
try {
await clearUsers()
console.log('Success')
} catch (e) {
console.error(e)
}
})()
If any function call within the clearUsers() function throws, for example await admin.save(), execution will stop on that line and return a rejected Promise which will be caught in the corresponding catch blocks in both variants.

Using Bluebird is a good option to handle callbacks in promises, It makes it easier to understand to an extent.
However I would recommend you to try
Async.js
It is built particularly for managing asynchronous nature of javascript.
This particular function of the library would do exactly what you want.

Replacing callbacks with promises in Node.js

I have a simple node module which connects to a database and has several functions to receive data, for example this function:
dbConnection.js:
import mysql from 'mysql';
const connection = mysql.createConnection({
host: 'localhost',
user: 'user',
password: 'password',
database: 'db'
});
export default {
getUsers(callback) {
connection.connect(() => {
connection.query('SELECT * FROM Users', (err, result) => {
if (!err){
callback(result);
}
});
});
}
};
The module would be called this way from a different node module:
app.js:
import dbCon from './dbConnection.js';
dbCon.getUsers(console.log);
I would like to use promises instead of callbacks in order to return the data.
So far I've read about nested promises in the following thread: Writing Clean Code With Nested Promises, but I couldn't find any solution that is simple enough for this use case.
What would be the correct way to return result using a promise?

Using the Promise class
I recommend to take a look at MDN's Promise docs which offer a good starting point for using Promises. Alternatively, I am sure there are many tutorials available online.:)
Note: Modern browsers already support ECMAScript 6 specification of Promises (see the MDN docs linked above) and I assume that you want to use the native implementation, without 3rd party libraries.
As for an actual example...
The basic principle works like this:
Your API is called
You create a new Promise object, this object takes a single function as constructor parameter
Your provided function is called by the underlying implementation and the function is given two functions - resolve and reject
Once you do your logic, you call one of these to either fullfill the Promise or reject it with an error
This might seem like a lot so here is an actual example.
exports.getUsers = function getUsers () {
// Return the Promise right away, unless you really need to
// do something before you create a new Promise, but usually
// this can go into the function below
return new Promise((resolve, reject) => {
// reject and resolve are functions provided by the Promise
// implementation. Call only one of them.
// Do your logic here - you can do WTF you want.:)
connection.query('SELECT * FROM Users', (err, result) => {
// PS. Fail fast! Handle errors first, then move to the
// important stuff (that's a good practice at least)
if (err) {
// Reject the Promise with an error
return reject(err)
}
// Resolve (or fulfill) the promise with data
return resolve(result)
})
})
}
// Usage:
exports.getUsers() // Returns a Promise!
.then(users => {
// Do stuff with users
})
.catch(err => {
// handle errors
})
Using the async/await language feature (Node.js >=7.6)
In Node.js 7.6, the v8 JavaScript compiler was upgraded with async/await support. You can now declare functions as being async, which means they automatically return a Promise which is resolved when the async function completes execution. Inside this function, you can use the await keyword to wait until another Promise resolves.
Here is an example:
exports.getUsers = async function getUsers() {
// We are in an async function - this will return Promise
// no matter what.
// We can interact with other functions which return a
// Promise very easily:
const result = await connection.query('select * from users')
// Interacting with callback-based APIs is a bit more
// complicated but still very easy:
const result2 = await new Promise((resolve, reject) => {
connection.query('select * from users', (err, res) => {
return void err ? reject(err) : resolve(res)
})
})
// Returning a value will cause the promise to be resolved
// with that value
return result
}

With bluebird you can use Promise.promisifyAll (and Promise.promisify) to add Promise ready methods to any object.
var Promise = require('bluebird');
// Somewhere around here, the following line is called
Promise.promisifyAll(connection);
exports.getUsersAsync = function () {
return connection.connectAsync()
.then(function () {
return connection.queryAsync('SELECT * FROM Users')
});
};
And use like this:
getUsersAsync().then(console.log);
or
// Spread because MySQL queries actually return two resulting arguments,
// which Bluebird resolves as an array.
getUsersAsync().spread(function(rows, fields) {
// Do whatever you want with either rows or fields.
});
Adding disposers
Bluebird supports a lot of features, one of them is disposers, it allows you to safely dispose of a connection after it ended with the help of Promise.using and Promise.prototype.disposer. Here's an example from my app:
function getConnection(host, user, password, port) {
// connection was already promisified at this point
// The object literal syntax is ES6, it's the equivalent of
// {host: host, user: user, ... }
var connection = mysql.createConnection({host, user, password, port});
return connection.connectAsync()
// connect callback doesn't have arguments. return connection.
.return(connection)
.disposer(function(connection, promise) {
//Disposer is used when Promise.using is finished.
connection.end();
});
}
Then use it like this:
exports.getUsersAsync = function () {
return Promise.using(getConnection()).then(function (connection) {
return connection.queryAsync('SELECT * FROM Users')
});
};
This will automatically end the connection once the promise resolves with the value (or rejects with an Error).

Node.js version 8.0.0+:
You don't have to use bluebird to promisify the node API methods anymore. Because, from version 8+ you can use native util.promisify:
const util = require('util');
const connectAsync = util.promisify(connection.connectAsync);
const queryAsync = util.promisify(connection.queryAsync);
exports.getUsersAsync = function () {
return connectAsync()
.then(function () {
return queryAsync('SELECT * FROM Users')
});
};
Now, don't have to use any 3rd party lib to do the promisify.

Assuming your database adapter API doesn't output Promises itself you can do something like:
exports.getUsers = function () {
var promise;
promise = new Promise();
connection.connect(function () {
connection.query('SELECT * FROM Users', function (err, result) {
if(!err){
promise.resolve(result);
} else {
promise.reject(err);
}
});
});
return promise.promise();
};
If the database API does support Promises you could do something like: (here you see the power of Promises, your callback fluff pretty much disappears)
exports.getUsers = function () {
return connection.connect().then(function () {
return connection.query('SELECT * FROM Users');
});
};
Using .then() to return a new (nested) promise.
Call with:
module.getUsers().done(function (result) { /* your code here */ });
I used a mockup API for my Promises, your API might be different. If you show me your API I can tailor it.

2019:
Use that native module const {promisify} = require('util'); to conver plain old callback pattern to promise pattern so you can get benfit from async/await code
const {promisify} = require('util');
const glob = promisify(require('glob'));
app.get('/', async function (req, res) {
const files = await glob('src/**/*-spec.js');
res.render('mocha-template-test', {files});
});

When setting up a promise you take two parameters, resolve and reject. In the case of success, call resolve with the result, in the case of failure call reject with the error.
Then you can write:
getUsers().then(callback)
callback will be called with the result of the promise returned from getUsers, i.e. result

Using the Q library for example:
function getUsers(param){
var d = Q.defer();
connection.connect(function () {
connection.query('SELECT * FROM Users', function (err, result) {
if(!err){
d.resolve(result);
}
});
});
return d.promise;
}

Below code works only for node -v > 8.x
I use this Promisified MySQL middleware for Node.js
read this article Create a MySQL Database Middleware with Node.js 8 and Async/Await
database.js
var mysql = require('mysql');
// node -v must > 8.x
var util = require('util');
// !!!!! for node version < 8.x only !!!!!
// npm install util.promisify
//require('util.promisify').shim();
// -v < 8.x has problem with async await so upgrade -v to v9.6.1 for this to work.
// connection pool https://github.com/mysqljs/mysql [1]
var pool = mysql.createPool({
connectionLimit : process.env.mysql_connection_pool_Limit, // default:10
host : process.env.mysql_host,
user : process.env.mysql_user,
password : process.env.mysql_password,
database : process.env.mysql_database
})
// Ping database to check for common exception errors.
pool.getConnection((err, connection) => {
if (err) {
if (err.code === 'PROTOCOL_CONNECTION_LOST') {
console.error('Database connection was closed.')
}
if (err.code === 'ER_CON_COUNT_ERROR') {
console.error('Database has too many connections.')
}
if (err.code === 'ECONNREFUSED') {
console.error('Database connection was refused.')
}
}
if (connection) connection.release()
return
})
// Promisify for Node.js async/await.
pool.query = util.promisify(pool.query)
module.exports = pool
You must upgrade node -v > 8.x
you must use async function to be able to use await.
example:
var pool = require('./database')
// node -v must > 8.x, --> async / await
router.get('/:template', async function(req, res, next)
{
...
try {
var _sql_rest_url = 'SELECT * FROM arcgis_viewer.rest_url WHERE id='+ _url_id;
var rows = await pool.query(_sql_rest_url)
_url = rows[0].rest_url // first record, property name is 'rest_url'
if (_center_lat == null) {_center_lat = rows[0].center_lat }
if (_center_long == null) {_center_long= rows[0].center_long }
if (_center_zoom == null) {_center_zoom= rows[0].center_zoom }
_place = rows[0].place
} catch(err) {
throw new Error(err)
}