We Keep Coding

return results from .then()

Ok, so I need to connect to a MySQL database through SSH and the connection works fine. I am able to execute queries with no problem. I can also get the results and print it out. The thing is, I need something simpler since I will have to send a lot of queries to this database. Below is the code for a promise which creates a connection to this database.
const SSHConnection = new Promise((resolve, reject) => {
sshClient.on('ready', () => {
sshClient.forwardOut(
forwardConfig.srcHost,
forwardConfig.srcPort,
forwardConfig.dstHost,
forwardConfig.dstPort,
(err, stream) => {
if (err) reject(err);
const updatedDbServer = {
...dbServer,
stream
};
const connection = mysql.createConnection(updatedDbServer);
connection.connect((error) => {
if (error) {
reject(error); // Return error
}
resolve(connection); // OK : return connection to database
});
});
}).connect(tunnelConfig);
});
Then I have this piece of code that gives me access to said connection and allows me to send queries. The problem is that I need the return value of my queries and be able to use it in other modules for my project. For example, export a single function to be used to send queries like sendQuery('Enter SQL here').
function sendQuery(sql) {
SSHConnection.then(
function(connection) {
connection.query(
sql,
function(err, results, fields) {
return results; // <---------- I want to return this from the function 'sendQuery()'
}
);
},
function(error) {
console.log("Something wrong happened");
}
);
}
I can work with the results inside SSHConnection.then() but this isn't functional for me.
I would like to have something similar below to work.
// Main function
(async function() {
let res = sendQuery(`SELECT 23+2 AS Sum;`);
console.log(res); // Outputs Sum: 25
})();
So to my question. Is there a way to access the results from a query inside of a promise.then(), from the outside?

I think the problem is you need to add another return statement to your code.
function sendQuery(sql) {
return SSHConnection.then(
function(connection) {
return connection.query(
sql,
function(err, results, fields) {
return results; // <---------- I want to return this from the function 'sendQuery()'
}
);
},
function(error) {
console.log("Something wrong happened");
}
);
}
This should return the results from the query properly IF connection.query returns a promise. I'm not sure if it does. If it does then you can just execute the function like so.
// Main function
(async function() {
let res = await sendQuery(`SELECT 23+2 AS Sum;`);
console.log(res); // Outputs Sum: 25
})();
If connection.query does not return a promise then I suppose you could wrap it in a promise like so.
function sendQuery (sql) {
return SSHConnection.then(
function (connection) {
return new Promise((resolve, reject) => {
connection.query(
sql,
function (err, results, fields) {
if (err)reject(err)
resolve(results) // <---------- I want to return this from the function 'sendQuery()'
}
)
})
},
function (error) {
console.log('Something wrong happened')
}
)
}

love the name by the way...really liked that movie. As to your question, I'd suggest a couple of things:
if you have a lot of queries coming, you might consider moving the connection independent of the query, so that the connection setup and teardown isn't part of the cost of time for the query itself. If you have a single connection, single DB, etc., then you could instantiate the connection once, at startup, and then leave the connection open, and reference it in your queries.
your question:
function sendQuery(sql) {
const resValues = await SSHConnection.then(
function(connection) {
connection.query(
sql,
function(err, results, fields) {
return results; // <---------- I want to return this from the function 'sendQuery()'
}
);
},
function(error) {
console.log("Something wrong happened");
}
);
return(resValues); // or handle error cases
}
Note I added a return value from the ".then()" call, which captures your "results" return value, and then returns that from the parent function (sendQuery)

Method does not wait for Promise to be resolved

I have a function that I'm trying to call and basically force it to wait for a response before continuing onto the next thing.
I have two functions, both are asynchronous.
The first one looks something like this, with all parameters that begin with an '_' to be used as callbacks:
async function formatJson(input, _sendToThirdParty, _handleLogs, _setDimensions)
{
...do some work here to format the payload
if(onlineConnectionRequired)
{
_setDimensions(itemToUpdate, object);
}
else {
// Do non-online based transformations here
}
...do more work after the above
}
Basically from that, I'm trying to call this method setDimensions which looks like this:
async function setDimensions(itemToUpdate, object) {
try
{
if(itemToUpdate != null)
{
console.log("Loading dimensions");
await Promise.resolve(function() {
ns.get(`inventoryItem/${object['Item ID']}?expandSubResources=true`)
.then((res) => {
console.log("Inventory Item Loaded. Updating dimensions...");
itemToUpdate.consignments.push(
{
consignmentID: object.conID,
barcode: object.barcode,
itemID: '', // leaving as empty for now
width : res.data.custitem_width,
length : res.data.custitem_length,
height : res.data.custitem_height,
weight : res.data.custitem_weight,
fragile: object.fragile === 'T' ? 1 : 0,
description: object.description
}
);
console.log("Dimensions Finalised");
})
});
}
}
catch(err)
{
console.log(err);
const message = `Error attempting to set the dimensions for ${object['Item ID']}`;
console.log(message);
throw new Error(message);
}
}
The problems I'm coming across are:
The code from the first method continues on before waiting for the promise to resolve, but I need it to wait so I can fully finish building up the payload before it continues on doing the next bits
If I try and include the await keyword before my call to _setDimensions(...) in the first method, I get an error "SyntaxError: await is only valid in async function", but I would've thought that it was an async function?
If anyone could help, that would be incredibly appreciated! Thank you!!

The correct design of your functions are below:
formatJson(input, (err, value) => {
if(err) {
// Error handler goes here
//console.log(err);
throw err;
} else {
// Implementation for working with returned value
console.log(value);
}
});
function formatJson(input, callback)
{
//...do some work here to format the payload
if(onlineConnectionRequired)
{
setDimensions(itemToUpdate, object)
.then((updatedItem) => {
// Implement anything here to work with your
// result coming from setDimensions() function
//console.log(updatedItem);
// Callback with null error and updatedItem as value
callback(null, updatedItem);
})
.catch((err) => {
// Callback with err object and null value
callback(err, null);
});
}
else {
// Do non-online based transformations here
}
//...do more work after the above
}
function setDimensions(itemToUpdate, object) {
try
{
if(inventoryItemID != null)
{
console.log("Loading dimensions");
return new Promise(function(resolve, reject) {
ns.get(`inventoryItem/${object['Item ID']}?expandSubResources=true`)
.then((res) => {
console.log("Inventory Item Loaded. Updating dimensions...");
itemToUpdate.consignments.push(
{
consignmentID: object.conID,
barcode: object.barcode,
itemID: '', // leaving as empty for now
width : res.data.custitem_width,
length : res.data.custitem_length,
height : res.data.custitem_height,
weight : res.data.custitem_weight,
fragile: object.fragile === 'T' ? 1 : 0,
description: object.description
}
);
console.log("Dimensions Finalised");
resolve(itemToUpdate);
})
.catch((err) => reject(err));
});
}
}
catch(err)
{
console.log(err);
const message = `Error attempting to set the dimensions for ${object['Item ID']}`;
console.log(message);
throw new Error(message);
}
}
Mistakes in your code:
Your formatJson function had async keyword but your formatJson function had callback functions named _sendToThirdParty, _handleLogs, _setDimensions. There are 3 types of implementation to create asynchronous codes. You can use callbacks, Promises or async/await. But Promises and async/await are the same except their uses cases and syntaxses. When you define a function as async fn() {...} it basically return a new Promise, so it is equal saying fn() { return new Promise(); }. Functions with callbacks have a shape like function(params, callback) { callback(cbParams); } you can use call callback function in a several branches in your function. But you only have single callback function, your code had 3 callback functions. Also note that functions with callback don't have async keyword. This is not valid because as I mentioned earlier, an async function will return a Promise. So you should not (but you can) define a function as async function(params, callback) like you did in your first method. This is definition is not wrong and it can work but it's not valid.
Your second method was an async function which was returning nothing. So I have changed it to normal function with returning a Promise.

Is the formatJson method beeing called inside an async method? It need to be, and before _setDimensions you need to add a await keyword.
And, as Daniel said, use the promise constructor.

Node.JS wait for each parallel mysql query to finish

I need assistance with the following: I have 4 MySQL queries which I wish to run in parallel but it is essential that all queries are completed before continuing in the main function; The four queries get different parts of the data (they are structured differently and access different tables) that is then used in the main function. Namely, I used async.parallel (even tried adding the await keyword) but still ended up continuing before the final callback is called in async.parallel. I even considered cluster and worker threads, but each query function is too different from one another.
The functions in async.parallel are MySQL connection queries in the form of:
connection.query(q, function(err, res) {
if(err) {
callback(err, [])
} else {
// do something
callback(null, res)
}
})
Each function is shaped differently and this is only a reduced form for the sake of brevity. Example of the function:
async main_function() {
await async.parallel({
"1": function(callback) { conn.query(q1, function(...) { console.log("1"); callback(null, 1) }) },
"2": function(callback) { conn.query(q2, function(...) { console.log("2"); callback(null, 2) }) },
"3": function(callback) { conn.query(q3, function(...) { console.log("3"); callback(null, 3) }) },
"4:" function(callback) { conn.query(q4, function(...) { console.log("4"); callback(null, 4) }) }
}, function(err, results) { /*final callback; do something*/ console.log("Finished"); })
console.log("Continuing");
//continue
}
Expected output:
3
2
4
1
Finished
Continuing
Actual output:
Continuing
3
2
4
1
Finished
If there's an entire other method (even without async.parallel), I'm open to it. I will likely have multiple distinct parallel MySQL queries later in the function and in other parts of the program, so having a parametrized form such as with async.parallel is extremely useful.
Javascript being single-threaded means that solutions like 'a while loop that checks whether the queries are finished' do not work. This main_function will be called very often so forking will cause issues.
EDIT: The Jakub's solution is rather simple as in my program works as the following:
async main_function() {
await Promise.all([
new Promise((resolve, reject) => {
connection.query(q1 , function (error, result) {
if (error != null) {
console.log("mysql query error", error);
return reject(error);
} else {
console.log("1");
resolve(result);
}
})
}),
new Promise((resolve, reject) => {
connection.query(q1 , function (error, result) {
if (error != null) {
console.log("mysql query error", error);
return reject(error);
} else {
console.log("2");
resolve(result);
}
})
}), ... // Two more times
]).then((values) => {
data[0] = values[0];
data[1] = values[1];
data[2] = values[2];
data[3] = values[3];
}).catch(function (err) { console.log(err); });
console.log("Continuing"); //continue
}
This is very crude right now, and if anybody wishes to suggest cleaner/better solutions for future programmers, go ahead.

These query functions are callback based (judging from your code), you can promisify them:
import { promisify } from 'utils';
async main_function() {
const res = await Promise.all([
promisify(conn.query)(q1),
promisify(conn.query)(q2),
// all your other queries
]);
console.log("Continuing");
//continue
}

How to wait for a loop creating streams to finish before saving?

I have a problem with knowing when the loop is finished,
app.post('/api/books', upload.array('images'), function(req, res) {
let book = req.body.book;
let arr = [];
// GridFS get connection with DB
var gfs = gridfsstream(conn.db);
var writestream;
for (i = 0; i < req.files.length; i++) {
writestream = gfs.createWriteStream({
filename: req.files[i].originalname
});
fs.createReadStream('uploads/' + req.files[i].filename).pipe(writestream);
writestream.on("close", function(file) {
console.log(file.filename + "stored successfully into mongodb using gridfs");
});
writestream.on("error", function(file) {
console.log(file.filename + "not stored into mongodb using gridfs");
});
base64(writestream.name, function(response) {
arr.push(response);
});
}
book.images = arr;
Book.addBook(book, function(err, book) {
if (err) {
throw err;
}
res.json(book);
});
});
The problem is: The array arr is empty when I do
book.images = arr
I need wait the for loop be finished , but how can i do that?
I know this works because i already put a console.log() and works correctly
base64(writestream.name, function(response) {
arr.push(response);
});

Probably best to use Promise.all here, but you need to wrap each of the "files" in a Promise and return resolved based on when the writeStream in each is completed or errors:
app.post('/api/books', upload.array('images'), function(req,res) {
let book = req.body.book;
var gfs = gridfsstream(conn.db);
Promise.all(
req.files.map(function(file) => {
return new Promise(function(resolve,reject) {
var writestream = gfs.createWriteStream({
filename: file.originalname
});
fs.createReadStream('uploads/'+file.filename).pipe(writestream);
writestream.on("error",reject);
writestream.on("close",function() {
base64(writestream.name, function(response) {
resolve(response);
});
});
})
})
)
.then(function(images) {
book.images = images;
Book.addBook(book,function(err,book) {
if (err) throw err; // or whatever
res.json(book)
});
})
.catch(function(err) => {
// Deal with errors
});
});
That involves no additional dependencies, however you could alternately use async.map as an additional dependency:
app.post('/api/books', upload.array('images'), function(req,res) {
let book = req.body.book;
var gfs = gridfsstream(conn.db);
async.map(
req.files,
function(file,callback) {
var writestream = gfs.createWriteStream({
filename: file.originalname
});
fs.createReadStream('uploads/'+file.filename).pipe(writestream);
writestream.on("error",callback);
writestream.on("close",function() {
base64(writestream.name, function(response) {
callback(null,response);
});
});
},
function(err,images) {
if (err) throw err;
book.images = images;
Book.addBook(book,function(err,book) {
if (err) throw err; // or whatever
res.json(book)
});
}
);
});
So they look pretty similar, and basically they are doing the same thing. In each case the "loop" is now a .map() which passes in the current filename as an argument and returns an array of the transformed response, which is in this case the output from your base64 function. They key here is that the resolve or callback is basically in control of when things happen.
In the case using Promise.all, the .map() is a basic JavaScript .map() function on the array, which is essentially returning an "array of Promise" which all implement the reject and resolve functions which are then called by the respective handlers on the stream.
Here the "Promises" all execute and return there results within the Promise.all and pass the output array to the block with .then(), which has the content and can then pass to your method to update/create as may be the case.
In the async.map example, this rather uses a callback argument which is again supplied to the event handlers on the stream. In exactly the same way, the final block receives the output or error, and can again pass to your method to persist the data.
There is a slight difference in the actual execution of calls, but both essentially apply the same principle that we "signal" output has completed to the mechanism providing the "loop", so that we know when "all" have completed.

Mongoose Find Results and Async

I am writing a NodeJS script that will run every hour through Heroku's scheduler. I am quering the Mongo instance I have (mongohq/compose) and then doing something with those results. I am working with Mongoose.js and the find() command. This returns an array of results. With those results I need to perform additional queries as well as some additional async processing (sending email, etc).
Long story short, due to node's async nature I need to wait until all the processing is complete before I call process.exit(). If I do not the script stops early and the entire result set is not processed.
The problem is that I have a christmas tree effect of calls at this point (5 nested asnyc calls).
Normally I'd solve this with the async.js library but I'm having a problem seeing this through with this many callbacks.
How can I make sure this entire process finishes before exiting the script?
Here's the code that I'm working with (note: also using lodash below as _):
Topic.find({ nextNotificationDate: {$lte: moment().utc()}}, function (err, topics) {
if (err) {
console.error(err);
finish();
} else {
_.forEach(topics, function (topic, callback) {
User.findById(topic.user, function (err, user) {
if (err) {
// TODO: impl logging
console.error(err);
} else {
// Create a new moment object (not moment.js, an actual moment mongoose obj)
var m = new Moment({ name: moment().format("MMM Do YY"), topic: topic});
m.save(function(err) {
if(err) {
// TODO: impl logging
console.error(err);
} else {
// Send an email via postmark
sendReminderTo(topic, user, m._id);
// Update the topic with next notification times.
// .. update some topic fields/etc
topic.save(function (err) {
if(err) {
console.error(err);
} else {
console.log("Topic updated.");
}
})
}
})
}
});
console.log("User: " + topic.user);
});
}
});

Part of what is making your code confusing is the usage of else statements. If you return your errors, you won't need the else statement and save 4 lines of indentation for every callback. That in and of itself will make things drastically more readable.
Using async:
Topic.find({nextNotificationDate: {$lte: moment().utc()}}, function (err, topics) {
if (err) {
console.error(err);
return finish(err);
}
async.each(topics, function(topic, topicCallback) {
async.auto({
user: function (callback) {
User.findById(topic.user, callback);
},
moment: function(callback) {
var m = new Moment({name: moment().format("MMM Do YY"), topic: topic});
m.save(callback);
},
topic: ["moment", "user", function (callback, results) {
var m = results.moment;
var user = results.user;
sendReminderTo(topic, user, m._id);
topic.save(callback);
}]
}, function(err) {
if (err) {
return topicCallback(err);
}
console.log("Topic updated.")
return topicCallback();
});
}, function(err) {
if (err) {
console.error(err);
return finish(err);
}
return finish();
});
});