We Keep Coding

Passing csvtojson converter's value to a variable outside of its scope

I'm currently fiddling around with Node.js and I stuck with this issue.
I'm using the csvtojson converter (https://github.com/Keyang/node-csvtojson) as a separate module that I can call in my other JS files as many times as I want.
Here is my tools.js:
module.exports = {
csvToJson: function (csvPath) {
var Converter = require('csvtojson').Converter;
var converter = new Converter({});
var transfer = "DEFAULT";
converter.fromFile(csvPath, function(err, result){
if (err) {
return console.log(err);
}
else {
transfer = result;
}
});
return transfer;
}
};
And here is how I call it:
var countriesCsvFile = path.join(__dirname, '..', 'testDataFiles', 'countries.csv');
//GRAB TOOLS
var tools = require('../app/tools');
console.log(tools.csvToJson(countriesCsvFile));
The result is always the "DEFAULT" value which indicates, that the converter is not touching it.
I want to pass it as the return value of the function, to further be able to process the data on the fly, without creating a file, and read that.
It is surely some scope issue, but after scratching my scalp for a few hours, and browsing the questions I couldn't retrieve anything remotely useful.
Also, another note: If I call console.log(result) instead of transfer = result, it shows me my precious and desired data.

You have to pass in a callback function because the csvToJson function is returning 'transfer' before any value is assigned to it. Like Sirko said, it's asynchronous. You can also use promises instead of callbacks but that's another topic in itself.
module.exports = {
csvToJson: function (csvPath, callback) {
var Converter = require('csvtojson').Converter;
var converter = new Converter({});
converter.fromFile(csvPath, function(err, result){
if (err) {
callback(err);
}
else {
callback(null, result);
}
});
}
};

How to return a nodejs callback as a number, from a JSON object property value?

I am trying to do maths on a number within a JSON object (the price of a stock ticker).
I want it to be a variable called 'btcusd_price', that I can then use to do arithmetic with.
How do I get a variable i can work with?
https://tonicdev.com/npm/bitfinex
var Bitfinex = require('bitfinex');
var bitfinex = new Bitfinex('your_key', 'your_secret');
var btcusd_price;
btcusd_price = bitfinex.ticker("btcusd", function(err, data) {
if(err) {
console.log('Error');
return;
}
console.log(data.last_price);
return data.last_price;
});
typeof btcusd_price;
console.log(btcusd_price); //i'm trying to work with the price, but it seems to be 'undefined'?

You have to set the values when they are available, the code is async and you were using the value before it is applied to btcusd_price. Note that the proper way to use the variable is when the callback executes its code.
Please, see the working code below:
Bitfinex = require('bitfinex');
var bitfinex = new Bitfinex('your_key', 'your_secret');
var btcusd_price;
bitfinex.ticker("btcusd", function(err, data) {
if(err) {
console.log('Error');
return;
}
btcusd_price = data.last_price;
console.log(typeof btcusd_price);
console.log(btcusd_price);
});

Functions that can't execute together

I was recently building a scraper module to get some information with nodejs until I encountered this "little" problem. The modules that I'm using are cheeriojs and request.
Actually the module works like a charm if I call only one method at a time. It contains three function and only two of them are exported, this is the code:
'use strict';
var request = require('request'),
cheerio = require('cheerio'),
counter = 0;
function find(term, cat, callback) {
// All the check for the parameters
scrape("http://.../search.php?search=" + encodeURIComponent(term), cat, callback);
}
function last(cat, callback) {
// All the check for the parameters
scrape("http://google.com/", cat, callback);
}
function scrape(url, cat, callback) {
request(url, function (error, response, body) {
if (!error && response.statusCode == 200) {
var $ = cheerio.load(body);
var result = [];
var items = $('.foo, .foo2').filter(function() {
// Condition to filter the resulted items
});
items.each(function(i, row) {
// Had to do another request inside here to scrape other information
request( $(".newpagelink").attr("href"), function(error, response, body) {
var name = $(".selector").text(),
surname = $(".selector2").text(),
link = cheerio.load(body)('.magnet').attr('href'); // This is the only thing that I'm scraping from the new page, the rest comes from the other "cheerio.load"
// Push an object in the array
result.push( { "name": name, "surname": surname, "link": link } );
// To check when the async requests are ended
counter++;
if(counter == items.length-1) {
callback(null, result);
}
});
});
}
});
}
exports.find = find;
exports.last = last;
The problem now, as I was saying, is that if I create a new node script "test.js" and I call only last OR find, it works perfectly! But if I call both the methods consecutively like this:
var mod = require("../index-tmp.js");
mod.find("bla", "blabla", function(err, data) {
if (err) throw err;
console.log(data.length + " find");
});
mod.last(function(err, data) {
console.log(data.length + " last");
});
The results are completely messed up, sometimes the script doesn't even print something, other times print the result of only "find" or "last", and other times returns a cheeriojs error (I won't add here to not mess you up, because probably it's my script's fault). I thought also to repeat the same function two times for both the methods but nothing, the same problems occur... I don't know what else to try, I hope you'll tell me the cause of this behavior!

Your counter variable is global, not specific to each scrape call. It wouldn't work if you called find twice at the same time either, or last.
Move the declaration and initialisation of var counter = 0; into the scrape function, or even better right next to the result and items declarations.

From scanning your code quickly, this is probably due to the variable counter being global. These are asynchronous functions, so they will both act on counter at the same thing. Move the declaration inside of the scrape function.
If you need more information about asynchronous programming, refer to Felix's great answer in this question.

MongoDB / Javascript Scope Issue

I'm connecting and making an insert with Node/MongoDB, but due to a scope issue, I can't access the connection from the function. Any idea how to make the 'db' variable global scope?
mongodb.connect("mongodb://localhost:27017/userDB", function(err, db) {
if(!err) {
console.log("We are connected");
} else {
console.log(err);
}
});
function RegisterUser(user, pass) {
var collection = db.collection('users');
var docs = [{username:user}, {password: pass}];
collection.insert(docs, {w:1}, function(err, result) {
collection.find().toArray(function(err, items) {});
socket.emit('message', items);
});
}
/var/www/baseball/app.js:80
var collection = db.collection('users'); <--db is not defined
^
ReferenceError: db is not defined
at RegisterUser (/var/www/baseball/app.js:80:20)
at ParseData (/var/www/baseball/app.js:63:6)

In general, only make the connection once, probably as you app starts up. The mongo driver will handle pooling etc., at least, so I was told by experts from MongoLabs. Note that this is very different than what you might do in Java, etc... Save the db value returned by connect() somewhere, perhaps a global or in app, or in a commonly used one of your modules. Then use it as needed in RegisterUser, DeleteUser, etc.

More of a node question really, but probably worth the tag since it gets asked a bit. So you say there is a scoping issue and you are right as the variable is local to the callback function on the .connect() method and is not visible anywhere else. One way is to dump all your logic inside that callback, so there is no scoping issue, but you probably don't want to do that.
Asking "how do I set a global", is also not really the right approach. Well not directly as there are general funny things about breaking up the "async" pattern of node. So a better approach is with some kind of "singleton" instance where you set the connection only once, but as that is global or can otherwise be "required" in for use in other areas of your application.
Here is one "trivial" approach to demonstrate, but there are many ways to do the same thing:
var async = require('async'),
mongodb = require('mongodb'),
MongoClient = mongodb.MongoClient;
var Model = (function() {
var _db;
var conlock;
return {
getDb: function(callback) {
var err = null;
if ( _db == null && conlock == null ) {
conlock = 1;
MongoClient.connect('mongodb://localhost/test',function(err,db) {
_db = db;
conlock == null;
if (!err) {
console.log("Connected")
}
callback(err,_db);
});
} else if ( conlock != null ) {
var count = 0;
async.whilst(
function() { return ( _db == null ) && (count < 5) },
function(callback) {
count++
setTimeout(callback,500);
},
function(err) {
if ( count == 5 )
err = new Error("connect wait exceeded");
callback(err,_db);
}
);
} else {
callback(err,_db);
}
}
};
})();
async.parallel(
[
function(callback) {
console.log("call model");
Model.getDb(function(err,db) {
if (err) throw err;
if (db != undefined)
console.log("db is defined");
callback();
});
},
function(callback) {
console.log("call model again");
Model.getDb(function(err,db) {
if (err) throw err;
if (db != undefined)
console.log("db is defined here as well");
callback();
})
}
],
function(err) {
Model.getDb(function(err,db) {
db.close();
});
}
);
So out little "Model" object here has a single method in .getDb(), and it also maintains a private variable holding the _db connection once it has been established. The basic logic on that method is to see if _db is defined and where it not then establish a connection with the driver. On the connection callback the _db variable is then set.
The other thing here is the method itself accepts a "callback" so this is how you use it later, where either an error or the current connection will be returned.
The last part is just a demonstration of two functions to be implemented in code. Where in the first call the call to connect to the database is made before following into the callback function provided.
The next time we call though, the connection is already set in the private variable, so that data is merely returned and you don't establish a connection again.
There are various ways to implement this sort of thing, but that is the basic logic pattern to follow. There are many other "helper" implementations that wrap the MongoDB driver to make these sort of things simple, as well as managing connection pools and ensuring the connection is up as well for you, so it may be well worth looking at these even if you are still insistent on doing all the work yourself from the lower level driver base.

First of all, you are only going to be registering users once you have a connection, so do what ever work you need to do in there... so call RegisterUser from within the connection scope. If you want to use the db object within that function you will need to pass in the parameters as db
RegisterUsers(db, user, pass)
you may then use db within the function

Best method to string together variety of DB calls in Node js

I basically need to make about 3 calls to get the data for a json object.. It basically JSON array of JSON object which have some attributes, one of which is an array of other values selected using a second query, then that one also has an array inside which is selected with another db call.
I tried using asyn.concatSeries so that I can dig down into the bottom call and put together all the information I collected for one root json object but that's creating a lot of unexpected behaviour..
Example of JSON
[
{
"item" : "firstDbCall"
"children" : [ {
"name" : "itemDiscoveredWithSecondDBCall"
"children" : [ itemsDiscoveredwith3rdDBCall]
},
]
}
]
This is really difficult using node.js. I really need to figure out how to do this properly since I have to do many of these for different purposes.
EDIT
This is the code i have. There's some strange behaviour with async.concatSeries. The results get called multiple times after each one of the functions finish for each array. So i had to put a check in place. I know it's very messy code but i've been just putting band-aids all over it for the past 2 hours to make it work..
console.log("GET USERS HAREDQARE INFO _--__--_-_-_-_-_____");
var query = "select driveGroupId from tasks, driveInformation where agentId = '"
+ req.params.agentId + "' and driveInformation.taskId = tasks.id order by driveInformation.taskId desc;";
connection.query(query, function(err, rows) {
if (rows === undefined) {
res.json([]);
return;
}
if(rows.length<1) { res.send("[]"); return;}
var driveGroupId = rows[0].driveGroupId;
var physicalQuery = "select * from drives where driveGroupId = " + driveGroupId + ";";
connection.query(physicalQuery, function(err, rows) {
console.log("ROWSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS");
console.log(rows);
async.concatSeries(rows, function(row, cb) {
console.log("-------------------------------SINGLE ROW-------------------------------------");
console.log(row);
if(row.hasLogicalDrives != 0) {
console.log("HAS LOGICAL DRIVES");
console.log(row.id);
var query = "select id, name from logicalDrives where driveId = " + row.id;
connection.query(query, function(error, drives) {
console.log("QUERY RETURNED");
console.log(drives);
parseDriveInfo(row.name, row.searchable, drives, cb);
});
}
else
var driveInfo = { "driveName" : row.name, "searchable" : row.searchable};
console.log("NO SUB ITEMS");
cb(null, driveInfo);
}, function(err, results) {
console.log("GEETTTTINGHERE");
console.log(results);
if(results.length == rows.length) {
console.log("RESULTS FOR THE DRIVE SEARCH");
console.log(results);
var response = {"id": req.params.agentId};
response.driveList = results;
console.log("RESPONSE");
console.log(response);
res.json(response);
}
});
});
});
};
parseDriveInfo = function(driveName, searchable, drives, cb) {
async.concatSeries(drives, function(drive,callback) {
console.log("SERIES 2");
console.log(drive);
console.log("END OF DRIVE INFO");
var query = "select name from supportedSearchTypes where logicalDriveId = " + drive.id;
connection.query(query, function(error, searchTypes) {
drive.searchTypes = searchTypes;
var driveInfo = { "driveName" :driveName,
"searchable" : searchable,
"logicalDrives" : drive
};
callback(null, driveInfo);
});
}, function (err, results) {
console.log("THIS IS ISISIS ISISISSISISISISISISISISISIS");
console.log(results);
if(results.length === drives.length) {
console.log("GOTHERE");
cb(null, results);
}
});
}

Getting good enough with async to use exactly the right combination of methods under the right circumstances takes a fair amount of experience. Most likely your case in particular can be handled with async.waterfall if its query1 then query2(dataFoundByQuery1) then query3(dataFoundByQuery2). But depending on the circumstances you need to mix and match async methods appropriately and sometimes have 2 levels - for example a "big picture" async.waterfall where some of the steps in the waterfall do async.parallel or async.series as needed. I've never used async.concat and given your needs I think you have chosen the wrong method. The workhorses are async.each, async.eachSeries, async.waterfall, and async.map, at least for the web app & DB query use cases I mostly encounter, so make sure you really have those understood before exploring the more specific convenience methods.

EDIT: This is a more in depth example based on use of the connection library you seem to be using. Please note, some of this is javascript psuedo code. Things like adding objects to the resultsArray are clearly not complete, the only thing I took time to make sure was correct is the "flow of logic" as it pertains to callbacks. Everything else is for you to implement. In order to support multiple calls to the same callback function and maintain state from call to call, the best way is to wrap the set of callbacks in a closure. This allows the callbacks to share some state with the main event loop. This allows you to pass arguments to the callbacks, without actually having to pass them as arguments, much like class variables in c++, or even globals in javascript, but we haven't poluted the global scope :)
function queryDataBase(query) {
//wrap the whole query in a function so the callbacks can share some
//variables with similar scope. This is called a closure
int rowCounter = 0;
var dataRowsFromStep2;
var resultsArray = {};
connection.query(query, dataBaseQueryStep2);
function dataBaseQueryStep2(err, rows) {
//do something with err and rows
dataRowsFromStep2 = rows;
var query = getQueryFromRow(dataRowsFromStep2[rowCounter++]);//Always zero the first time. Might need to double check rows isn't empty!
connection.query(query, dataBaseQueryStep3);
}
function dataBaseQueryStep3(err, rows) {
//do something with err and rows
if(rowCounter < dataRowsFromStep2.size) {
resultsArray.add(rows);//Probably needs to be more interesting, but you get the idea
//since this is within the same closure, rowCounter maintains it's state
var query = getQueryFromRow(dataRowsFromStep2[rowCounter++]);
//recursive call query using dataBaseQueryStep3 as it's callback repeatedly until
//we run out of rows to call it on.
connection.query(query, dataBaseQueryStep3)
} else {
//when the if statement fails we have no more rows to run queries on so return to main program flow
returnToMainProgramLogic(resultsArray);
}
}
}
function returnToMainProgramLogic(results) {
//continue running your program here
}
I personally like the above logic better than the syntax async produces... I believe the heart of your problem rests in your nested calls to async, and the fact that ASYN itself, runs the series of functions asynchronously, but in order(confusing I know). If you write your program like this, you won't have to worry about it!

I would strongly suggest using sequelize.js It provides a really powerful orm that allows you to chain queries together. It also allows you to directly load your data into js objects, write dynamic sql, and connect to many different databases. Picture ActiveRecord from the Ruby world for Node.