We Keep Coding

Database call in a function without callback with Node.js and Postgresql

I'm trying out the framework node.js on one of my projects.
I'm really seeing some good advantages on what they called "event-driven, non-blocking I/O model" however if my project there are some moments where I don't necessarily want to have some asynchronous calls and to be able to several operation before launching some asynchronous call.
Especially when I want to do some factorization and create some functions.
Typically I have the following case:
I know that in several part of my program I have to check if a media is existing in my database for a given string or id.
So as a guy who tried to stay organize I want to create a function that I will call each time I need to check this.
However, I did not find the way to do that with node.js and pg (the npm PostgreSQL library (https://github.com/brianc/node-postgres/) . Indeed, there is always a callback in the function and the return is null because of the callback. Here is an example below
/*
Function which is supposed to check if a media existing
*/
function is_media_existing (url_or_id){
log.debug("is_media_existing : begin of the function", {"Parameter" : url_or_id});
pg.connect(connectionstring, function (err, client, done) {
if (err) {
log.warning("is_media_existing : Problem with Database connection", {
"Parameter": url_or_id,
"Error": err
});
}
if (isNaN(url_or_id)) {
// Case is parameter is not a number (string)
var query = client.query('SELECT COUNT(*) as count FROM media WHERE url = $1::string ', url_or_id);
query.on('error', function (error) {
log.warning("is_media_existing : Problem with Database query (connection to db passed but not query " +
"", {"Parameter": url_or_id, "Error": error});
});
return query;
} else {
// Case is parameter is a int
log.debug("is_media_existing : Type of Parameter is a string");
// Case is parameter is not a number (string)
var query = client.query('SELECT COUNT(*) as count FROM media WHERE id = $1::id ', url_or_id);
query.on('error', function (error) {
log.warning("is_media_existing : Problem with Database query (connection to db passed but not query " +
"", {"Parameter": url_or_id, "Error": error});
});
return query;
}
});
}
// Executing the function
var test = is_media_existing("http://random_url_existing_in_db");
// test is always null as the return is in a callback and the callback is asynchronous
i have the feeling my question is touching the core concepts of node.js, and perhaps my approach is wrong and I apologize in advance.
I know it's not good to wait for a response before doing something.
But what's the alternative? How can I factorize my code into functions when I need some functionalities in several part of my code?
So if there would be anyone who could explain how to do that with a best practice of programming it would be great.
Thanks
Anselme

As Cody says, you probably dont want to do synchronous function.
The way you should handle the situation in your example is to pass in your own callback like this
function is_media_existing (url_or_id, callback){
and then instead of return query; use your callback like this-
callback(query);
or probably better to follow the node convention for callback functions to have two parameters (err, result) so your callback would look like this
callback(null, query);
Here is a rework of your sample
function is_media_existing (url_or_id, callback){ /* callback(err, result) */
log.debug("is_media_existing : begin of the function", {"Parameter" : url_or_id});
pg.connect(connectionstring, function (err, client, done) {
if (err) {
done(err);
log.warning("is_media_existing : Problem with Database connection", {
"Parameter": url_or_id,
"Error": err
});
return callback(err, null);
/* note that this return is simply used to exit the connect's callback and the return value is typically
* not used it is the call to callback() that returns the error value */
}
var qrystr;
if (isNaN(url_or_id)) {
log.debug("is_media_existing : Type of Parameter is a string");
qrystr = 'SELECT COUNT(*) as count FROM media WHERE url = $1::string;';
} else {
qrystr = 'SELECT COUNT(*) as count FROM media WHERE id = $1::id;';
}
client.query(qrystr, [url_or_id], function(err, result){
done();
if(err){
/* .. */
}
callback(err, result);
});
});
}
// Executing the function
var test = is_media_existing("http://random_url_existing_in_db", function(err, result){
if(err){
}else {
}
});
If you end up with a hard nest of callbacks, promises are really worth looking into.

I don't think you really do want a synchronous call. The problem with synchronous calls in node is that it stops the entire process from doing anything while a synchronous function is running as it will stop the event loop. As an example lets say your sync function takes 2 seconds to complete. Your server will then do nothing for 2 full seconds. That 2 seconds includes everything (accepting new connections, everything else, etc). The reason blocking functions don't exist is because they are (very) bad. Here is an example how your function will react in an async manor.
is_media_existing("http://random_url_existing_in_db", function(exists){
if (exists){
//do stuff
} else {
//do this other stuff
}
});
Then within is_media_existing you will need to call that callback function when your query completes.
//psuedo
function is_media_existing(url, callback){
query('select COUNT(*) as count FROM media WHERE id = $1::id '. [url], function(err, result){
if (err)
callback(false)
else
callback(result.count > 0)
})
}

With the new ES6 plus async stuff and babel its simpler. You can npm i -g babel npm i babel-runtime then compile and run the following with babel test.js --optional runtime --stage 2 | node. Please read the following example carefully to see how to adapt it to your use case:
let testData = [
{ id: 0, childIds: [1,2]},
{ id: 1, childIds:[] }
];
function dbGet(ids) {
return new Promise( r=> {
// this an example; you could do any db
// query here and call r with the results
r(ids.map((id) => { return testData[id];}));
});
}
async function dbExists(ids) {
let found = await dbGet(ids);
return (found && found.length>0);
}
async function test() {
var exists = await dbExists([0]);
console.log(exists);
}
test().then(f=>{}).catch( e=> {console.log('e',e)});

node.js asynchronous passing variable between functions

i am taking my first steps with node.js and i came across this issue with passing variable in asynchronous way. I have this piece of code im using to create Facebook user:
req.tmpPassport = {};
var fb = new fbgraph.Facebook(accessToken, 'v2.2');
function initUser() {
fb.me(function (err, me) {
req.tmpPassport.me = me;
console.log(req.tmpPassport.me) // works
});
}
console.log(req.tmpPassport.me) // not working -> undefined var
i tried to figure out why the second log isn't working and i ended up reading about synchronous and asynchronous functions, so in attempt to implement what i read i tried coming up with a solution using callbacks, but no success.
my last attempt was this:
req.tmpPassport = {};
var fb = new fbgraph.Facebook(accessToken, 'v2.2');
function initUser() {
fb.me(function (err, me) {
req.tmpPassport.me = me;
});
fb.my.events(function (err, events) {
//console.log(events);
req.tmpPassport.events = events;
});
fb.my.friends(function (err, result) {
req.tmpPassport.results = result;
});
}
function passUser(){
console.log(req.tmpPassport);
return req.tmpPassport;
}
cp.exec(initUser, passUser);
but its not working...
what i am actually trying to achieve its to render this object with my express router var which looks like this:
router.get('/welcome', securePages, function(req, res, next){
res.render('welcome', {title:'Welcome to aDating', user:req.tmpPassport});
})
but i cant figure out how to pass this object only after created...any help please?

A method of chaining function calls when certain async tasks are done is one way to deal with this.
Looking at the first snippet of code, it would be rewritten as follows:
req.tmpPassport = {};
var fb = new fbgraph.Facebook(accessToken, 'v2.2');
function initUser() {
fb.me(function (err, me) {
console.log(req.tmpPassport.me) // works
req.tmpPassport.me = me;
// triggers execution of the next step
post_populating_passport();
});
}
function post_populating_passport() {
// this function is only executed after the callback from the async call
console.log(req.tmpPassport.me);
}

Node.js is an asynchronous programming language, a fundamental concept in its core. You either need to use function callbacks (not a good practice) or available npms utilities for async flow.

function done in javascript

Actually I am reading some codes in Nodejs, but I cannot get what it means because some common use of function that I don't quite understand. Can someone tell me what the function(done) means? It is a callback or something? in the js file I can't see the definition of done. Thanks
Andes
var getRedisUri = exports.getRedisUri = function (done) {
if (process.env.CLUSTER_CONFIG_URI) {
return done(null, process.env.CLUSTER_CONFIG_URI);
}
//redis uri
Fs.readFile(
'/opt/redis_uri',
{encoding: 'utf8'},
function (err, redis_uri_data) {
if (err) {return done(err);}
var redis_uri = redis_uri_data.toString();
if (process.env.NODE_ENV !== 'production' &&
process.env.VM_DOMAIN &&
(redis_uri.indexOf('localhost') !== -1 || redis_uri.indexOf('127.0.0.1') !== -1)) {
redis_uri = redis_uri.replace('localhost', process.env.VM_DOMAIN);
redis_uri = redis_uri.replace('127.0.0.1', process.env.VM_DOMAIN);
}
done(null, redis_uri);
});
};

That line is just the start of a function definition. function(done) just means that when this function is called, it should be called with one argument, and during the execution of the function, that argument will be referred to as done.
More specifically, I think it has to do with middleware (are you using express.js here?). If you have a bunch of middleware functions in express, the express router will call those functions for you, and pass in a function as an argument which, when called, passes the request to the next middleware function. It seems like that's what's happening here, as done gets called at the end of the function.
Node.js has stuff like this because it's all asynchronous, so you can't use return statements for much.

Writing express.js app. Where do helper methods go?

So I have started with express.js - my first JS web dev framework. I didn't make anything small, but started a bigger project. I'm learning, and building at the same time.
Coming from a Python/Flask background, express seems very complicated.
Like in python, if I want a helper method, I can just put it on top of the file, or in a new module, and import it. Super easy. But in node/express, things are async, and everything is in exports or module.exports (??). Where do helper methods go? How do I call them with callbacks?
In another question I asked, I was doing the same kind of computation multiple times. In Python, I would write a method (with if statements and parameters), and call it multiple times, using a for.. in... loop. The code I have right now is very redundant.
How do I do it in express? What are best practices for writing express code?

It really depends of what your helper is doing. If it operates with data which is passed as a parameter to it then you may save it in an external module and use require to access it.
// helpers/FormatString.js
module.exports = function(str) {
return str.toUpperCase();
}
// app.js
var formatter = require("./helpers/FormatString");
However, if you need to modify the request or the response object then I'll suggest to define it as a middleware. I.e.:
app.use(function(req, res, next) {
// ... do your stuff here
});

#Krasimir gave a correct answer. Regarding your question how to deal with asynchronous helper functions I can give you an example (not the usual foo/bar, but one of my own helper functions):
var cached; // as modules can act like singletons I can share values here
module.exports = function fetchGoogleCerts(options, callback) { // <-- callback
var now = Date.now();
if (ttl > now && cached) {
callback(null, cached); // <-- return with success
return;
}
request({
uri: options.certsUrl || defaultCertsUrl,
strictSSL: true
}, function (err, response, body) {
var error, certs;
if (!err && response.statusCode === 200) {
certs = jsonParse(body); // a local function
if (!certs) {
callback('parse_error', null); // <-- return an error
return;
}
cached = certs;
// ... more code
callback(null, cached); // <-- success case
} else {
error = {
error: err,
statusCode: response.statusCode
};
log.error(error, 'error fetching google certs');
callback(error); // <-- return an error
}
});
}
};
And using the helper:
fetchGoogleCerts = require('./google-certs.js'),
module.exports = function fetchCertsAndDecodeIdToken(token, options, callback) {
fetchGoogleCerts(options, function (err, googleCerts) {
if (err) {
callback({
errorCode: codes.io_error,
errorMsg: 'Unable to fetch Google certificates',
error: err
});
return;
}
decodeAndVerifyGoogleIdToken(googleCerts, token, options, callback);
});
};
As you can see above, the simple solution is to provide a callback function to your asynchronous helper function.
Of course you can also export an Object which extends EventEmitter, then you might not need a callback function, but register for the events. Here is an Example for a helper which emits events.

How could I optimize this code with too many nest functions?

I'm writing a Chrome extension with the socket api(though this doc is out of date, the latest version of the api is here), and I found that the code is really hard to organize:
All the methods are under the namespace chrome.experimental.socket, I would just use socket below for simplicity.
socket.create("tcp", {}, function(socketInfo){
var socketId = socketInfo.socketId;
socket.connect(socketId, IP, PORT, function(result){
if(!result) throw "Connect Error";
socket.write(socketId, data, function(writeInfo){
if(writeInfo.bytesWritten < 0) throw "Send Data Error";
socket.read(socketId, function(readInfo){
if(readInfo.resultCode < 0) throw "Read Error";
var data = readInfo.data; // play with the data
// then send the next request
socket.write(socketId, data, function(writeInfo){
socket.read(socketId, function(readInfo){
// ............
});
});
});
})
});
})
because both socket.write and socket.read are asynchronous, I have to nest the callbacks to make sure that the next request is send after the previous request got the correct response.
it's really hard to manage these nested functions, how could I improve it?
UPDATE
I'd like to have a method send which I can use as:
send(socketId, data, function(response){
// play with response
});
// block here until the previous send get the response
send(socketId, data, function(response){
// play with response
});

How about (something like) this?
var MySocket = {
obj: null,
data: null,
start: function() { ... some code initializing obj data, ending with this.create() call },
create: function() { ... some code initializing obj data, ending with this.connect() call },
connect: function() { ... some connection code, ending with this.write() call },
write: function() { ... some writing code that updates this.data, ending with this.read() call },
read: function() { ... you probably get the idea at this point )) ... },
};
This object could be used with MySocket.start() or something. The idea is to encapsulate all data (and nested calls) within the single (yet more-o-less globally usable) object.
Or even more, one can create two objects: one purely for writing purposes, and another for purely reading, each operating with its own data, then wrap them (and their inter-calls, so to speak) into a single SocketManager object.

Consider using an asynchronous continuation passing style, where functions end with a SetInterval call with the function they were passed. Then we construct a function that entwines two functions to call each other using this mechanism. The guts of it would be like this:
var handle;
// pairs two functions
function pair(firstfunc, secondfunc, startarg) {
var callbackToFirst = function(valuetofill) {
handle = setInterval(firstfunc(valuetofill,callbackToSecond));
};
var callbackToSecond = function(valuetofill) {
handle = setInterval(secondfunc(valuetofill,callbackToFirst));
};
callbackToFirst(startarg);
}
What we are doing here is constructing a pair of mutually-calling callbacks which take a single argument, which each contain references to the two inter-calling functions. We then kick off the process by calling the first callback.
Construct the pair for an example pair of read and write functions (assuming you've set the socketId in the enclosing object definition):
// starts read/write pair, sets internal variable 'handle' to
// interval handle for control
function startReadWrite(initialarg, myDataFunc) {
var readcall = function(value, func) {
readSocket(getData(myDataFunc(func)));
};
var writecall = function(value, func) {
writeSocket(checkBytesWritten(func));
};
handle = pair(readcall, writecall, initialarg);
}
The rest of the object is like this:
function myIO() {
var socketInfo, socketId, handle;
function create(func) {
socket.create('tcp',{},function(thisSocketInfo) {
socketInfo = thisSocketInfo;
}
setInterval(func(this),0);
}
function connect(IP, PORT, func) {
socket.connect(p_socketId, IP, PORT, function() {
if(!result) throw "Connect Error";
setInterval(func(result),0);
});
}
function readSocket(func) {
socket.read(p_socketId, function(readInfo){
setInterval(func(readInfo),0);
});
}
function writeSocket(data, func) {
socket.write(p_socketId, data, function(writeInfo){
setInterval(func(writeInfo),0)
});
}
function checkBytesWritten(writeInfo, func) {
if(writeInfo.bytesWritten < 0) throw "Send Data Error";
setInterval(func(writeInfo),0);
}
function getData(readInfo, func) {
if(readInfo.resultCode < 0) throw "Read Error";
var data = readInfo.data;
setInterval(func(data),0);
}
//** pair and startReadWrite go here **//
}
Finally the call to set the whole thing going:
var myIOobj = new myIO();
myIOobj.create(startReadWrite(myDataFunc));
Notes:
This is meant to demonstrate a style, not be ready code! Don't just copy and paste it.
No, I haven't tested this; I do javascript but not Chrome API stuff yet. I'm focussing on the callback mechanisms etc.
Be careful with the different classes of callback; single argument callbacks (like the read and write callbacks) which take a single value (as presumably defined by the API), and 2 argument callbacks (like most of the methods) which take an argument and a function to call at the end.
The getData method takes a callback and passes data to it; this callback (myDataFunc) is the function that actually gets to use the data. It needs to take a callback as a second argument and call it synchronously or asynchronously.
TLDR: Consider using asynchronous calls to avoid the nesting. I've given a vague example of a mechanism to have two functions call each other continuously using this style as seems to be needed.
Although I call it asynchonous, the setInterval calls will execute serially, but the key is that the stack is cleared after the parent call is done, rather than adding endless layers with nesting.