We Keep Coding

Database call in a function without callback with Node.js and Postgresql

I'm trying out the framework node.js on one of my projects.
I'm really seeing some good advantages on what they called "event-driven, non-blocking I/O model" however if my project there are some moments where I don't necessarily want to have some asynchronous calls and to be able to several operation before launching some asynchronous call.
Especially when I want to do some factorization and create some functions.
Typically I have the following case:
I know that in several part of my program I have to check if a media is existing in my database for a given string or id.
So as a guy who tried to stay organize I want to create a function that I will call each time I need to check this.
However, I did not find the way to do that with node.js and pg (the npm PostgreSQL library (https://github.com/brianc/node-postgres/) . Indeed, there is always a callback in the function and the return is null because of the callback. Here is an example below
/*
Function which is supposed to check if a media existing
*/
function is_media_existing (url_or_id){
log.debug("is_media_existing : begin of the function", {"Parameter" : url_or_id});
pg.connect(connectionstring, function (err, client, done) {
if (err) {
log.warning("is_media_existing : Problem with Database connection", {
"Parameter": url_or_id,
"Error": err
});
}
if (isNaN(url_or_id)) {
// Case is parameter is not a number (string)
var query = client.query('SELECT COUNT(*) as count FROM media WHERE url = $1::string ', url_or_id);
query.on('error', function (error) {
log.warning("is_media_existing : Problem with Database query (connection to db passed but not query " +
"", {"Parameter": url_or_id, "Error": error});
});
return query;
} else {
// Case is parameter is a int
log.debug("is_media_existing : Type of Parameter is a string");
// Case is parameter is not a number (string)
var query = client.query('SELECT COUNT(*) as count FROM media WHERE id = $1::id ', url_or_id);
query.on('error', function (error) {
log.warning("is_media_existing : Problem with Database query (connection to db passed but not query " +
"", {"Parameter": url_or_id, "Error": error});
});
return query;
}
});
}
// Executing the function
var test = is_media_existing("http://random_url_existing_in_db");
// test is always null as the return is in a callback and the callback is asynchronous
i have the feeling my question is touching the core concepts of node.js, and perhaps my approach is wrong and I apologize in advance.
I know it's not good to wait for a response before doing something.
But what's the alternative? How can I factorize my code into functions when I need some functionalities in several part of my code?
So if there would be anyone who could explain how to do that with a best practice of programming it would be great.
Thanks
Anselme

As Cody says, you probably dont want to do synchronous function.
The way you should handle the situation in your example is to pass in your own callback like this
function is_media_existing (url_or_id, callback){
and then instead of return query; use your callback like this-
callback(query);
or probably better to follow the node convention for callback functions to have two parameters (err, result) so your callback would look like this
callback(null, query);
Here is a rework of your sample
function is_media_existing (url_or_id, callback){ /* callback(err, result) */
log.debug("is_media_existing : begin of the function", {"Parameter" : url_or_id});
pg.connect(connectionstring, function (err, client, done) {
if (err) {
done(err);
log.warning("is_media_existing : Problem with Database connection", {
"Parameter": url_or_id,
"Error": err
});
return callback(err, null);
/* note that this return is simply used to exit the connect's callback and the return value is typically
* not used it is the call to callback() that returns the error value */
}
var qrystr;
if (isNaN(url_or_id)) {
log.debug("is_media_existing : Type of Parameter is a string");
qrystr = 'SELECT COUNT(*) as count FROM media WHERE url = $1::string;';
} else {
qrystr = 'SELECT COUNT(*) as count FROM media WHERE id = $1::id;';
}
client.query(qrystr, [url_or_id], function(err, result){
done();
if(err){
/* .. */
}
callback(err, result);
});
});
}
// Executing the function
var test = is_media_existing("http://random_url_existing_in_db", function(err, result){
if(err){
}else {
}
});
If you end up with a hard nest of callbacks, promises are really worth looking into.

I don't think you really do want a synchronous call. The problem with synchronous calls in node is that it stops the entire process from doing anything while a synchronous function is running as it will stop the event loop. As an example lets say your sync function takes 2 seconds to complete. Your server will then do nothing for 2 full seconds. That 2 seconds includes everything (accepting new connections, everything else, etc). The reason blocking functions don't exist is because they are (very) bad. Here is an example how your function will react in an async manor.
is_media_existing("http://random_url_existing_in_db", function(exists){
if (exists){
//do stuff
} else {
//do this other stuff
}
});
Then within is_media_existing you will need to call that callback function when your query completes.
//psuedo
function is_media_existing(url, callback){
query('select COUNT(*) as count FROM media WHERE id = $1::id '. [url], function(err, result){
if (err)
callback(false)
else
callback(result.count > 0)
})
}

With the new ES6 plus async stuff and babel its simpler. You can npm i -g babel npm i babel-runtime then compile and run the following with babel test.js --optional runtime --stage 2 | node. Please read the following example carefully to see how to adapt it to your use case:
let testData = [
{ id: 0, childIds: [1,2]},
{ id: 1, childIds:[] }
];
function dbGet(ids) {
return new Promise( r=> {
// this an example; you could do any db
// query here and call r with the results
r(ids.map((id) => { return testData[id];}));
});
}
async function dbExists(ids) {
let found = await dbGet(ids);
return (found && found.length>0);
}
async function test() {
var exists = await dbExists([0]);
console.log(exists);
}
test().then(f=>{}).catch( e=> {console.log('e',e)});

Node JS MySQL query function not returning result

I am running a MySQL query inside a .js file running on Node JS. I have the connection setup ok and the query works but when I try returning the result back to the original call it doesn't seem to work.
function sqlQuery(query){
console.log("Running query");
var conn = connection();
var result = false;
conn.query(query, function(err, rows, fields) {
conn.end();
if (!err){ result = rows; } else { result = err; }
});
return result;
}
var loginResult = sqlQuery("SELECT * FROM `players`");
console.log(loginResult);
If I use the following code it does write the result to the console inside the query but not the final "loginResult". I am not getting any errors so my question is - is there an error in the way I am getting the returned result?
if (!err){ result = rows; console.log(rows); } else { result = err; }

Virtually everything in Node.js is asynchronous, and SQL query functions definitely are. You're calling conn.query(query, callback), which means that query is called, and then once there is a result at some point in the future, your callback function gets called with the result for you to work with. So:
conn.query(query, function runThisEventually(err, rows, fields) {
if (err) {
console.error("One or more errors occurred!");
console.error(err);
return;
}
processResults(rows, fields);
});
You won't get the result immediately after calling conn.query(...), so your code gets to do "other things" in the mean time, and at some point, your callback will be triggered and you can pick up result processing there.

Node.js: Asynchronous callback confusion

I am trying to figure out how to create asynchronous functions for a web app. I am doing a database query, manipulating the data to a format that is more convenient, and then trying to set my router to pass back that file.
//Module 1
//Module 1 has 2 functions, both are necessary to properly format
function fnA(param1){
db.cypherQuery(query, function(err, result){
if(err){
return err;
}
var reformattedData = {};
//code that begins storing re-formatted data in reformattedData
//the function that handles the rest of the formatting
fnB(param1, param2);
});
});
function fnB(param1, reformattedData){
db.cypherQuery(query, function(err, result){
if(err){
return err;
}
//the rest of the reformatting that uses bits from the second query
return reformattedData;
});
});
exports.fnA = fnA;
Then in my router file:
var db = require('module1');
router.get('/url', function(req,res,next){
db.fnA(param1, function(err, result){
if (err){
return next(err);
}
res.send(result);
});
});
When I tried to test this out by hitting the URL indicated by the router, it just loads indefinitely.
I know what I have above is wrong, since I never wrote my function to require a callback. When I tried figuring out how to rewrite it though, I got really confused - How do I write my function to have a callback when the asynchronous stuff happens inside it?
Can someone help me rewrite my functions to use callbacks correctly, so that when I actually use the function, I can still work with the asynchronous response?

You use db.fa from your router file, and pass the second parameter as a callback function. but the function signature don't have the cb param and doesnt use it.
The main idea - you try to initiate an async operation and cannot know when it ends, so you send it a callback function to get triggered when all operations are done.
Fixed code should be like that:
//Module 1
//Module 1 has 2 functions, both are necessary to properly format
function fnA(param1, cb1){
db.cypherQuery(query, function(err, result){
if(err){
cb1(err); <-- return error to original call
}
var reformattedData = {};
//code that begins storing re-formatted data in reformattedData
//the function that handles the rest of the formatting
fnB(param1, param2, cb1);
});
});
function fnB(param1, reformattedData, cb1){
db.cypherQuery(query, function(err, result){
if(err){
cb1(err); <-- return error to original call
}
//the rest of the reformatting that uses bits from the second query
cb1(false, dataObjectToSendBack); <--- This will call the anonymouse function in your router call
});
});
exports.fnA = fnA;
Router file:
var db = require('module1');
router.get('/url', function(req,res,next){
db.fnA(param1, function(err, result){ <-- This anonymous function get triggered last
if (err){
return next(err);
}
res.send(result);
});
});

Extract data from variables inside multiple callbacks in javascript

I'm ok with javascript and callbacks, but I'm getting really annoyed at this and need to call on the the world of stackoverflow for help.
I have written a function, to be used in the following way:
var meth = lib.funcTion(a,b); // meth should hold an array of properties { c, d } once executed
So now inside lib.js, we have a structure like:
exports.funcTion = function (a,b) {
database.connect(params, function(err,get){
get.query(querylang, function(err, results){
var varsIwantToReturn = { var1: results[i].foo, var2: results[i].bar };
});
});
// Now how do i return 'varsIwantToReturn'?
};
I have seen some things about incorporating callback() into the function, but I'm not exactly sure how this works. I've also seen some people use exec() - again, im not sure on how or why to use it.
Please help :) thanks in advance.

Well, it's all asynchronous so if you attempt to return it - it'll return undefined. In JavaScript (Sans the new yield keyword) functions execute from top to bottom synchronously. When you make an IO call like a database call - it still executes synchronously. In fact- when varIwantToReturn gets population the function has long run and terminated.
What is left is to do the same thing async functions like database.connect and get.query do and have the function take a callback:
exports.function = function (a,b, callback) {
database.connect(params, function(err,get){
if(err) return callback(err, null); // don't suppress errors
get.query(querylang, function(err, results){
if(err) return callback(err, null); // don't suppress errors
var varsIwantToReturn = { var1: results[i].foo, var2: results[i].bar };
callback(null, varsIwantToReturn);
});
});
};
Then you'd call it like
myExportedFunction(myA,myB, function(err, resp){
if(err) recoverFromError(err);
// varsIWantToReturn are contained in `resp`
});

return value from asynchronous function in Nodejs

I am using nodejs to query data from Mongodb throught Mongoose.
After get the data, I want do something on that data before responding it to client. But I can not get the return-value. After looking on Google, I have learned Node.js functions is asynchronous javascript function (non I/O blocking). I try this tut (http://www.youtube.com/watch?v=xDW9bK-9pNY) but it is not work. Below is my code. The myObject is valued inside "find()" function and undefined outside "find()" function. So what should I do to get the data? Thanks!
var Person = mongoose.model('Person', PersonSchema);
var Product = mongoose.model('Product', ProductSchema);
var myObject = new Object();
Person.find().exec(function (err, docs) {
for (var i=0;i<docs.length;i++)
{
Product.find({ user: docs[i]._id},function (err, pers) {
myObject[i] = pers;
console.log(myObject[i]); //return the value is ok
});
console.log(myObject[i]); //return undefined value
}
console.log(myObject); //return undefined value
});
console.log(myObject); //return undefined value
app.listen(3000);
console.log('Listening on port 3000');

The reason you're getting undefined values is because the find function is asynchronous, and can finish at any time. In your case, it is finishing after you're using console.log(), so the values are undefined when you're accessing them.
To fix this problem, you can only use the values inside the find function's callback. It would look something like this:
var Person = mongoose.model('Person', PersonSchema);
var Product = mongoose.model('Product', ProductSchema);
var myObject = new Object();
function getData(docs, callback) {
function loop(i) {
Product.find({ user: docs[i]._id}, function (err, pers) {
myObject[i] = pers;
if (i < docs.length) {
loop(i + 1);
} else {
callback();
}
});
};
loop(0);
};
Person.find().exec(function(err, docs) {
getData(docs, function() {
// myObject has been populated at this point
});
});
The data processing has been moved to a loop that waits for the previous iteration to complete. This way, we can determine when the last callback has fired in order to fire the callback in the wrapper function.

Keep in mind that by the time the console.log functions are executed, the query has not yet finished, thus will display "undefined". That's the essence of nodeJS's asynchronicity.
For example,
Person.find().exec(function (err, docs) {
for (var i=0;i<docs.length;i++)
{
Product.find({ user: docs[i]._id},function (err, pers) {
myObject[i] = pers;
console.log(myObject[i]); //return the value is ok
});
console.log(myObject[i]); //return undefined value
}
console.log(myObject); //return undefined value
});
console.log(myObject); // <-- Initially, this value will be undefined. After some miliseconds (Or the duration of the .exec function, myObject will contain the results.
If you want to actually wait until the query is finished so you can use the values, I would recommend moving the app.listen(3000); and console.log('Listening on port 3000'); into the function's final callback.
I'd also recommend you to check out this node module. It will help you build asynchronous / synchronous functions more easily, and allow you to execute a callback when all the asynchronous functions are finished.