We Keep Coding

Waiting for pool.query inside a callback in Express

I'm using crypto.generateKeyPair inside a post endpoint in Express.
I need to insert the key generated inside my DB and then return within the endpoint the id row of the inserted row.
The endpoint code reads as:
app.post('/createKeys', (req, res) => {
crypto.generateKeyPair('rsa',,
(err, publicKey, privateKey) => {
if(!err) {
let id = myfunction(publicKey.toString('hex'),
privateKey.toString('hex'));
console.log(id)
} else {
res.status(500).send(err);
}
});
});
async function myfunction(publicKey, privateKey) {
await pool.query('INSERT INTO users (publickey, privatekey) VALUES ($1, $2) RETURNING id',
[publicKey, privateKey],
(error, results) => {
if (error) {
throw error;
}
resolve(results.rows[0]['id']);
});
};
However, inside the callback in crypto I get only a Promise, or undefined if I don't use async/await. How can I await myfunction result so I can send back to the user the id?

Several issues here:
await only does something useful when you are awaiting a promise.
pool.query() does not return a promise when you pass it a callback, so your await there is not doing anything useful.
resolve() is not a function that exists outside the context of creating a new promise with new Promise((resolve, reject) => { code here })
throw error inside an asynchronous callback will not do anything useful as there is no way to catch that exception and thus no way to implement any decent error handling. Don't write code that way. When you promisify the function (as shown below), you can then reject the promise and that will offer a way to propagate the error back to the caller.
Your choices for waiting for pool.query() with await here are:
Use the version of your database that natively supports promises and then don't pass a callback to pool.query() so that it returns a promise that tells you when it's complete.
Promisify your own function by wrapping pool.query() in a new promise and call resolve() and reject() appropriately.
Remember, do NOT mix plain callbacks and promise. Instead, promisify any asynchronous functions that use plain callbacks and then do all your logic flow with promises.
Here's a manually promisified version of your myfunction():
function myfunction(publicKey, privateKey) {
return new Promise((resolve, reject) => {
pool.query('INSERT INTO users (publickey, privatekey) VALUES ($1, $2) RETURNING id',
[publicKey, privateKey],
(error, results) => {
if (error) {
reject(error);
return;
}
resolve(results.rows[0]['id']);
});
});
}
crypto.generateKeyPairP = util.promisify(crypto.generateKeyPair);
app.post('/createKeys', async (req, res) => {
try {
const {publicKey, privateKey } = await crypto.generateKeyPairP('rsa');
const id = await myfunction(publicKey.toString('hex'), privateKey.toString('hex'));
console.log(id);
// send your response here, whatever you want it to be
res.send(id);
} catch(e) {
res.status(500).send(e);
}
});
Note that in this implementation, the resolve and reject functions come from the new Promise() - they don't just exist in this air.
But, there is likely a version of your database or an interface in your existing database module where pool.query() can return a promise directly using built-in promise support.

async await not working properly in mongo query

I have a project collection. Under the project schema have a property called tasks which is an array with the ids of the tasks. Now I am trying to write an api for singleProject, which should include the project details with a property taskDetails which will contain the detail of the tasks under the project. And for getting the task details I am using the ids which I stored in the task property under project. Below is the code I am using :
exports.getSingleProject = async (req, res) => {
req.project.logo = undefined;
await req.project.tasks.map((taskItem, index) => {
taskSchema.find({ _id: taskItem.toString() }).exec((err, task) => {
req.project.tasksDetails.push(task);
});
});
responseMessages(
res,
200,
true,
"Single project fetched successfully.",
req.project
);
};
but in the response getting taskDetails as an blank array. I guess I am using async await in wrong place. Need some help on this. Thanks in advance

The await keyword only works with a promise: an array is not a promise, but you can map to a promise and wait for all promises to resolve using Promise.all - see example below:
const tasks = await Promise.all(req.project.tasks.map(item => new Promise((accept, reject) => {
taskSchema.find({ _id: taskItem.toString() }).exec((err, task) => {
if(err) {
reject(err);
return;
}
accept(task);
});
});
The tasks variable would be an array of your tasks, which you can return in the response.

UnhandledPromiseRejectionWarning: Unhandled promise rejection in _.map in NodeJs

I always get the error when I run the code block below -
UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag --unhandled-rejections=strict (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
module.exports = (app, spotifyAPI) => {
app.get('/api/search', requireLogin, async (req, res) => {
const URI_BASE = keys.ComputerVisionEndpoint + 'vision/v3.0/analyze';
const imageUrl = "https://upload.wikimedia.org/wikipedia/commons/3/3c/Shaki_waterfall.jpg"; // will be sent as req body
var results;
// making API call to microsoft cognitive services API
try {
results = await axios({
method: 'post',
url: URI_BASE,
headers: {
'Content-Type': 'application/json',
'Ocp-Apim-Subscription-Key' : keys.ComputerVision
},
params: {
'visualFeatures': 'Tags',
'details': '',
'language': 'en'
},
data: {
"url": imageUrl,
}
});
} catch (err) {
return res.status(400).send(err);
}
// remove the common ones - indoor, outdoor, ground, wall, person, woman, man, ceiling, floor
const to_filter = results['data']['tags'];
_.remove(to_filter, (item) => {
return (item.name === 'indoor' || item.name === 'outdoor' || item.name === 'ground' || item.name === 'wall'
|| item.name === 'person' || item.name === 'woman' || item.name === 'man' || item.name === 'ceiling'
|| item.name === 'floor'
);
});
// searching for relevant songs and adding them to the playlist
var id;
try {
id = await search_and_add(req, res, spotifyAPI, to_filter, playlist_id);
} catch (err) {
if (err['statusCode'] === 401) {
req.logout();
return res.redirect('/');
}
else {
return res.status(400).send(err);
}
}
});
}
search_and_add = async (req, res, spotifyAPI, to_filter, playlist_id) => {
_.map(to_filter, async (tag) => {
try {
const song_details = await spotifyAPI.searchTracks(tag.name, { limit: 1 });
//const song_uri = song_details['body']['tracks']['items'][0]['id'];
console.log(song_details);
} catch (err) {
throw err;
}
});
return;
// figure out where to re direct user
};
I'm pretty sure it is because of the map statement in the search_and_add function, but I do not know how to get rid of it and provide the same functionality to make the try-catch block work? Could someone help?

You're not doing anything with the promises created by the callback to the _.map(…) call in search_and_add. They just get ignored, are not awaited, and will cause the warning when getting rejected. Presumably you meant to use Promise.all there?
function search_and_add(req, res, spotifyAPI, to_filter, playlist_id) {
return Promise.all(to_filter.map(async (tag) => {
// ^^^^^^^^^^^^
const song_details = await spotifyAPI.searchTracks(tag.name, { limit: 1 });
//const song_uri = song_details['body']['tracks']['items'][0]['id'];
console.log(song_details);
});
}

You are using map function since you need to wrap with Promise.allSettled function.
Promise.allSettled is available in node js 12 version and above.
if you are using less than node 12 then you need to use Promise.all
Promise.allSettled():
The Promise.allSettled() method returns a promise that resolves after all of the given promises have either fulfilled or rejected, with an array of objects that each describes the outcome of each promise.
It is typically used when you have multiple asynchronous tasks that are not dependent on one another to complete successfully, or you'd always like to know the result of each promise.
Promise.all():
The Promise.all() method takes an iterable of promises as an input, and returns a single Promise as an output. This returned promise will resolve when all of the input's promises have resolved and non-promises have returned, or if the input iterable contains no promises. It rejects immediately upon any of the input promises rejecting or non-promises throwing an error, and will reject with this first rejection message / error.
It is typically used when there are multiple asynchronous tasks that are dependent on one another to complete successfully, as it does not wait and will reject immediately upon any of the input promises rejecting.
refer this:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/allSettled
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all
https://sung.codes/blog/2019/05/18/promise-race-vs-promise-any-and-promise-all-vs-promise-allsettled/
const search_and_add = (req, res, spotifyAPI, to_filter, playlist_id) {
return Promise.allSettled(to_filter.map(async (tag) => {
const song_details = await spotifyAPI.searchTracks(tag.name, { limit: 1 });
//const song_uri = song_details['body']['tracks']['items'][0]['id'];
console.log(song_details);
return song_details;
}).catch(function(err){
console.log(err);
return err;
});
}
For error handling in async await:
Async Await is essentially syntactic sugar for promises, and if an await statement errors it will return a rejected promise, Instead of adding try -catch every where we can write a helper function that wraps our express routes to handle all rejected promises of route.
const asyncMiddleware = fn =>
(req, res, next) => {
Promise.resolve(fn(req, res, next))
.catch(next);
};
then wrap your route function like this
router.get('/users/:id', asyncMiddleware(async (req, res, next) => {
/*
if there is an error thrown in getUserFromDb, asyncMiddleware
will pass it to next() and express will handle the error;
*/
const user = await getUserFromDb({ id: req.params.id })
res.json(user);
}));
Note: You can also use npm package async-middleware for this.

Is creating a new promise with a async function call bad practice?

Snippets are from a node.js and mongoDB CRUD application.Github repo for full code. The code is working fine but unsure if my structure and use of promises and async await are bad practice.
handlers._newbies = {};
handlers._newbies.post = (parsedReq, res) => {
const newbie = JSON.parse(parsedReq.payload);
databaseCalls.create(newbie)
.then((result) => {
res.writeHead(200,{'Content-Type' : 'application/json'});
const resultToString = JSON.stringify(result.ops[0]);
res.write(resultToString);
res.end();
})
.catch(err => console.log(err));
};
const databaseCalls = {};
databaseCalls.create = (newbie) => {
return new Promise(async (resolve, reject) => {
try {
const client = await MongoClient.connect('mongodb://localhost:27017', { useNewUrlParser: true });
console.log("Connected correctly to server");
const db = client.db('Noob-List');
const result = await db.collection('newbies').insertOne(newbie);
client.close();
resolve(result);
} catch(err) {
console.log(err);
}
});
};
When the node server gets a POST request with the JSON payload, it calls the handlers._newbies.post handler which takes the payload and passed it to the
const newbie = JSON.parse(parsedReq.payload);
databaseCalls.create(newbie)
call. I want this database call to return a promise that holds the result of the db.collection('newbies').insertOne(newbie);
call. I was having trouble doing this with just returning the promise returned by the insertOne because after returning I cant call client.close();.
Again maybe what I have done here is fine but I haven't found anything online about creating promises with promises in them. Thank you for your time let me know what is unclear with my question.

It is considered an anti-pattern to be wrapping an existing promise in a manually created promise because there's just no reason to do so and it creates many an opportunities for error, particular in error handling.
And, in your case, you have several error handling issues.
If you get an error anywhere in your database code, you never resolve or reject the promise you are creating. This is a classic problem with the anti-pattern.
If you get an error after opening the DB, you don't close the DB
You don't communicate back an error to the caller.
Here's how you can do your .create() function without the anti-pattern and without the above problems:
databaseCalls.create = async function(newbie) {
let client;
try {
client = await MongoClient.connect('mongodb://localhost:27017', { useNewUrlParser: true });
console.log("Connected correctly to server");
const db = client.db('Noob-List');
return db.collection('newbies').insertOne(newbie);
} catch(err) {
// log error, but still reject the promise
console.log(err);
throw err;
} finally {
// clean up any open database
if (client) {
client.close();
}
}
}
Then, you would use this like:
databaseCalls.create(something).then(result => {
console.log("succeeded");'
}).catch(err => {
console.log(err);
});
FYI, I also modified some other things:
The database connection is closed, even in error conditions
The function returns a promise which is resolved with the result of .insertOne() (if there is a meaningful result there)
If there's an error, the returned promise is rejected with that error
Not particularly relevant to your issue with promises, but you will generally not want to open and close the DB connection on every operation. You can either use one lasting connection or create a pool of connections where you can fetch one from the pool and then put it back in the pool when done (most DBs have that type of feature for server-side work).

Replacing callbacks with promises in Node.js

I have a simple node module which connects to a database and has several functions to receive data, for example this function:
dbConnection.js:
import mysql from 'mysql';
const connection = mysql.createConnection({
host: 'localhost',
user: 'user',
password: 'password',
database: 'db'
});
export default {
getUsers(callback) {
connection.connect(() => {
connection.query('SELECT * FROM Users', (err, result) => {
if (!err){
callback(result);
}
});
});
}
};
The module would be called this way from a different node module:
app.js:
import dbCon from './dbConnection.js';
dbCon.getUsers(console.log);
I would like to use promises instead of callbacks in order to return the data.
So far I've read about nested promises in the following thread: Writing Clean Code With Nested Promises, but I couldn't find any solution that is simple enough for this use case.
What would be the correct way to return result using a promise?

Using the Promise class
I recommend to take a look at MDN's Promise docs which offer a good starting point for using Promises. Alternatively, I am sure there are many tutorials available online.:)
Note: Modern browsers already support ECMAScript 6 specification of Promises (see the MDN docs linked above) and I assume that you want to use the native implementation, without 3rd party libraries.
As for an actual example...
The basic principle works like this:
Your API is called
You create a new Promise object, this object takes a single function as constructor parameter
Your provided function is called by the underlying implementation and the function is given two functions - resolve and reject
Once you do your logic, you call one of these to either fullfill the Promise or reject it with an error
This might seem like a lot so here is an actual example.
exports.getUsers = function getUsers () {
// Return the Promise right away, unless you really need to
// do something before you create a new Promise, but usually
// this can go into the function below
return new Promise((resolve, reject) => {
// reject and resolve are functions provided by the Promise
// implementation. Call only one of them.
// Do your logic here - you can do WTF you want.:)
connection.query('SELECT * FROM Users', (err, result) => {
// PS. Fail fast! Handle errors first, then move to the
// important stuff (that's a good practice at least)
if (err) {
// Reject the Promise with an error
return reject(err)
}
// Resolve (or fulfill) the promise with data
return resolve(result)
})
})
}
// Usage:
exports.getUsers() // Returns a Promise!
.then(users => {
// Do stuff with users
})
.catch(err => {
// handle errors
})
Using the async/await language feature (Node.js >=7.6)
In Node.js 7.6, the v8 JavaScript compiler was upgraded with async/await support. You can now declare functions as being async, which means they automatically return a Promise which is resolved when the async function completes execution. Inside this function, you can use the await keyword to wait until another Promise resolves.
Here is an example:
exports.getUsers = async function getUsers() {
// We are in an async function - this will return Promise
// no matter what.
// We can interact with other functions which return a
// Promise very easily:
const result = await connection.query('select * from users')
// Interacting with callback-based APIs is a bit more
// complicated but still very easy:
const result2 = await new Promise((resolve, reject) => {
connection.query('select * from users', (err, res) => {
return void err ? reject(err) : resolve(res)
})
})
// Returning a value will cause the promise to be resolved
// with that value
return result
}

With bluebird you can use Promise.promisifyAll (and Promise.promisify) to add Promise ready methods to any object.
var Promise = require('bluebird');
// Somewhere around here, the following line is called
Promise.promisifyAll(connection);
exports.getUsersAsync = function () {
return connection.connectAsync()
.then(function () {
return connection.queryAsync('SELECT * FROM Users')
});
};
And use like this:
getUsersAsync().then(console.log);
or
// Spread because MySQL queries actually return two resulting arguments,
// which Bluebird resolves as an array.
getUsersAsync().spread(function(rows, fields) {
// Do whatever you want with either rows or fields.
});
Adding disposers
Bluebird supports a lot of features, one of them is disposers, it allows you to safely dispose of a connection after it ended with the help of Promise.using and Promise.prototype.disposer. Here's an example from my app:
function getConnection(host, user, password, port) {
// connection was already promisified at this point
// The object literal syntax is ES6, it's the equivalent of
// {host: host, user: user, ... }
var connection = mysql.createConnection({host, user, password, port});
return connection.connectAsync()
// connect callback doesn't have arguments. return connection.
.return(connection)
.disposer(function(connection, promise) {
//Disposer is used when Promise.using is finished.
connection.end();
});
}
Then use it like this:
exports.getUsersAsync = function () {
return Promise.using(getConnection()).then(function (connection) {
return connection.queryAsync('SELECT * FROM Users')
});
};
This will automatically end the connection once the promise resolves with the value (or rejects with an Error).

Node.js version 8.0.0+:
You don't have to use bluebird to promisify the node API methods anymore. Because, from version 8+ you can use native util.promisify:
const util = require('util');
const connectAsync = util.promisify(connection.connectAsync);
const queryAsync = util.promisify(connection.queryAsync);
exports.getUsersAsync = function () {
return connectAsync()
.then(function () {
return queryAsync('SELECT * FROM Users')
});
};
Now, don't have to use any 3rd party lib to do the promisify.

Assuming your database adapter API doesn't output Promises itself you can do something like:
exports.getUsers = function () {
var promise;
promise = new Promise();
connection.connect(function () {
connection.query('SELECT * FROM Users', function (err, result) {
if(!err){
promise.resolve(result);
} else {
promise.reject(err);
}
});
});
return promise.promise();
};
If the database API does support Promises you could do something like: (here you see the power of Promises, your callback fluff pretty much disappears)
exports.getUsers = function () {
return connection.connect().then(function () {
return connection.query('SELECT * FROM Users');
});
};
Using .then() to return a new (nested) promise.
Call with:
module.getUsers().done(function (result) { /* your code here */ });
I used a mockup API for my Promises, your API might be different. If you show me your API I can tailor it.

2019:
Use that native module const {promisify} = require('util'); to conver plain old callback pattern to promise pattern so you can get benfit from async/await code
const {promisify} = require('util');
const glob = promisify(require('glob'));
app.get('/', async function (req, res) {
const files = await glob('src/**/*-spec.js');
res.render('mocha-template-test', {files});
});

When setting up a promise you take two parameters, resolve and reject. In the case of success, call resolve with the result, in the case of failure call reject with the error.
Then you can write:
getUsers().then(callback)
callback will be called with the result of the promise returned from getUsers, i.e. result

Using the Q library for example:
function getUsers(param){
var d = Q.defer();
connection.connect(function () {
connection.query('SELECT * FROM Users', function (err, result) {
if(!err){
d.resolve(result);
}
});
});
return d.promise;
}

Below code works only for node -v > 8.x
I use this Promisified MySQL middleware for Node.js
read this article Create a MySQL Database Middleware with Node.js 8 and Async/Await
database.js
var mysql = require('mysql');
// node -v must > 8.x
var util = require('util');
// !!!!! for node version < 8.x only !!!!!
// npm install util.promisify
//require('util.promisify').shim();
// -v < 8.x has problem with async await so upgrade -v to v9.6.1 for this to work.
// connection pool https://github.com/mysqljs/mysql [1]
var pool = mysql.createPool({
connectionLimit : process.env.mysql_connection_pool_Limit, // default:10
host : process.env.mysql_host,
user : process.env.mysql_user,
password : process.env.mysql_password,
database : process.env.mysql_database
})
// Ping database to check for common exception errors.
pool.getConnection((err, connection) => {
if (err) {
if (err.code === 'PROTOCOL_CONNECTION_LOST') {
console.error('Database connection was closed.')
}
if (err.code === 'ER_CON_COUNT_ERROR') {
console.error('Database has too many connections.')
}
if (err.code === 'ECONNREFUSED') {
console.error('Database connection was refused.')
}
}
if (connection) connection.release()
return
})
// Promisify for Node.js async/await.
pool.query = util.promisify(pool.query)
module.exports = pool
You must upgrade node -v > 8.x
you must use async function to be able to use await.
example:
var pool = require('./database')
// node -v must > 8.x, --> async / await
router.get('/:template', async function(req, res, next)
{
...
try {
var _sql_rest_url = 'SELECT * FROM arcgis_viewer.rest_url WHERE id='+ _url_id;
var rows = await pool.query(_sql_rest_url)
_url = rows[0].rest_url // first record, property name is 'rest_url'
if (_center_lat == null) {_center_lat = rows[0].center_lat }
if (_center_long == null) {_center_long= rows[0].center_long }
if (_center_zoom == null) {_center_zoom= rows[0].center_zoom }
_place = rows[0].place
} catch(err) {
throw new Error(err)
}