I'm building a program that queries MySQL databases, gets the tables, fields, field data types, and entries and returns it as a single object to be later used to view the MySQL data as a table.
This is what the built object will look like:
{
`Table_Name`: {
Title: `Table_Name`,
Fields: {
`Field Name`: `Datatype`
},
RowData: []
}
}
The query to get the tables is fine, however the query to get the row data isn't. The query function looks like this:
function getRows(){
let secondpromises = [];
secondpromises.push(
new Promise((resolve, reject) => {
for(x in Tables){
Connect_SQL(SQLcreds, w_newSconn, (conn) => {
conn.query(`SELECT * FROM ${Tables[x]}`, (err, results) => {
if(err){
console.log(err);
reject(err);
}else{
for(r in results){
Tables[`${Tables[x].Title}`].RowData.push(results[r]);
}
resolve(results);
}
});
});
if(x == Tables.length - 1){
Promise.all(secondpromises).then(() => {
if(w_newSconn){
w_newSconn.close();
w_newSconn = null;
}
console.log(Tables);
});
}
}
})
);
}
The error is coming from conn.query(). It is throwing an error stating there is an error in my SQL syntax at:
SELECT * FROM [object Object]
I understand the reason why and I'm sure there is a way to resolve this through JSON.Stringify() but there must be a simpler way. I have already tried creating a variable like so:
let objArray = Object.keys(Tables)
But it still returned [object Object], any help would be appreciated.
Tables[x] is an object. You need to get the table name from it.
conn.query(`SELECT * FROM ${Tables[x].Title}`, (err, results) => {
It also looks like the property name is the same as the title, so you can do:
conn.query(`SELECT * FROM ${x}`, (err, results) => {
I ended up creating a variable in the loop
let table = keys[x]
and that did the trick, for whatever reason ${keys[x]} was returning undefined but the variable returned the table name. Theoretically I could have changed the for loops to a
for(x in Tables)
and x would have returned the title so I may go back and rewrite it that way. Thank you.
i have a simple question and i have read a lot of same issues here, but these are not exact the same or doesn't work for me :-(
I have a REST function called "addevent". The function gets a json input (req) and iterate through the json array to get some IDs to store them in an extra Array. That works perfect!
After that, the function should search in a mongodb for every single id and store some extra informations from this ID (e.g. the stored URL of this ID). With "console.log(result.link)" it works again perfect. But my problem is that, that i need to store this link in an extra Array (urlArray).
So how can i save the result of collection.findone(). I read something about, that findone() doesn't return a document, but a cursor? what does that mean? How do i have to handle that in my case?
That's the code:
exports.addevent = function(req, res) {
var ids = req.body;
var pArray = new Array();
var urlArray = new Array();
var eventName = ids.name;
for(var i in ids.photos) {
photoArray.push(ids.photos[i]);
var id = ids.photos[i]._id;
var collection = db.get().collection('photos');
collection.findOne({'_id':new mongo.ObjectID(id)},function(err, result) {
console.log(result.link);
}
)
}
Many thanks!
-------------------- Update --------------------
Ok, i think that has something to do with the asynch Callbacks. I found an article, but i don't know how to implement it in my case.
http://tobyho.com/2011/11/02/callbacks-in-loops/
And something about "promises" in javascript.
You can save the result of your search doing something like:
var foundPhoto = collection.find({_id':new mongo.ObjectID(id)}, function(err, photo){
if(!err){
return photo;
} else {
console.log(err)
return null;
}
});
This way you get the return statement of your query in the "photo" variable.
Background Information
I'm trying to write some javascript / node.js code that will do the following:
Query a redis database and get back a bunch of keys from my hash. This is what the command looks like on the redis-cli:
127.0.0.1:6379> hkeys widgets:1231231234
1) "13:00:00_17:00:00_mtwrf"
2) "00:00:00_00:00:00"
3) "08:00:00_12:00:00_mtwrf"
or
127.0.0.1:6379> hkeys widgets:2222222222
1) "00:00:00_00:00:00"
Each widget will have at least one key... called the default key. Default keys look like this: "00:00:00_00:00:00"
For each HKEYS query, if there's more than one result returned, I need to test each key returned (except the default) against a set of match criteria. Whichever key matches, is what is used to do a subsequent HGET against redis. So for example, in the first result set shown above... If key 3 was a match I would run the following command:
127.0.0.1:6379> hget widgets:2222222222 08:00:00_12:00:00_mtwrf
"some value"
127.0.0.1:6379>
If neither key 1 or 3 matches, then I query for the default key.
Code
I've recently discovered the async module. I'm currently using it in my code to loop through the results from HKEYS.
Please see the code below:
async.map Code
router.get('/:widgetnum', function(req, res, next) {
//validate widgetnum format
var widgetnum = req.params.widgetnum;
if ( !valid_widget(widgetnum) ) {
var retval = {"res":false, "msg":"malformed widgetnum"};
res.send(JSON.stringify(retval));
} else {
var keys = {};
redis.hkeys("widgets:" + widgetnum, function(err, data){
if (err) {
res.send(JSON.stringify(false));
}
if (data) {
current = getCurrentUTC(); // needed by iterator for match criteria. Using a global variable for now.
async.map(data, hash_iterator, function (err, iterator_results) {
if (iterator_results) {
console.log("it are: " + iterator_results);
}
res.send(iterator_results);
}); //end async map
}
}); //end redis.hkeys
}
});
Question / Problem
This is working in the sense that for each key returned by HKEYS, I'm able to run the "hash_iterator" function.
However, once inside the iterator function, after evaluation / running my match criteria on each result, I don't have all the information I need to run the secondary HGET query.
The code:
async.map(data, hash_iterator, function (err, iterator_results)
passes just the values:
"13:00:00_17:00:00_mtwrf"
"00:00:00_00:00:00"
"08:00:00_12:00:00_mtwrf"
But I need the hash name (in this case "widgets:" ) and the widgetnum to make the HGET call.
I guess I can use global variables... but I wanted to make sure that my approach in general is correct here.
Any input would be appreciated.
Here's the hash_iterator logic:
var hash_iterator = function (redis_key, doneCallBack) {
var results = {};
console.log("processing..." + redis_key);
//skip if default rule....
if (redis_key.indexOf('00:00:00_00:00:00') == -1){
//need to write logic here to do HGET and save
//results somewhere... in case no other keys match
} else {
// run some logic to test match criteria.
if (matchfound) {
console.log("bingo. found match in " + redis_key);
redis.hget(hashname + widgetnum + redis_key, function (e, results){
if (results) {
return doneCallBack(null, results);
} else {
return doneCallBack(null, null);
}
});
}else {
console.log ("no match");
}
}
return results;
}
You can solve it in functional style with map. Just generate array of pairs and then pass it to async.map, for example
from [1,2,3] you can generate [['key', '1'], ['key', '2'], ['key', 3]]
So here the code, that generating pairs:
redis.hkeys("widgets:" + widgetnum, function(err, data){
// some code goes here
var pairs = data.map(function(ts) {
return [`widgets:${widgetnum}`, ts];
});
// async.map call goes here
}); //end redis.hkey
Inside async.map iterator callback you can access key with iterator_results[0]
But in your case the variable widgetnums is not global variable. It's just variable from upper scope and you can use it inside nested functions if you define it in propper scope. It's very useful trick that called lexical closure.
I am trying to do maths on a number within a JSON object (the price of a stock ticker).
I want it to be a variable called 'btcusd_price', that I can then use to do arithmetic with.
How do I get a variable i can work with?
https://tonicdev.com/npm/bitfinex
var Bitfinex = require('bitfinex');
var bitfinex = new Bitfinex('your_key', 'your_secret');
var btcusd_price;
btcusd_price = bitfinex.ticker("btcusd", function(err, data) {
if(err) {
console.log('Error');
return;
}
console.log(data.last_price);
return data.last_price;
});
typeof btcusd_price;
console.log(btcusd_price); //i'm trying to work with the price, but it seems to be 'undefined'?
You have to set the values when they are available, the code is async and you were using the value before it is applied to btcusd_price. Note that the proper way to use the variable is when the callback executes its code.
Please, see the working code below:
Bitfinex = require('bitfinex');
var bitfinex = new Bitfinex('your_key', 'your_secret');
var btcusd_price;
bitfinex.ticker("btcusd", function(err, data) {
if(err) {
console.log('Error');
return;
}
btcusd_price = data.last_price;
console.log(typeof btcusd_price);
console.log(btcusd_price);
});
I'm using mongoose to insert some data into mongodb. The code looks like:
var mongoose = require('mongoose');
mongoose.connect('mongo://localhost/test');
var conn = mongoose.connection;
// insert users
conn.collection('users').insert([{/*user1*/},{/*user2*/}], function(err, docs) {
var user1 = docs[0], user2 = docs[1];
// insert channels
conn.collection('channels').insert([{userId:user1._id},{userId:user2._id}], function(err, docs) {
var channel1 = docs[0], channel2 = docs[1];
// insert articles
conn.collection('articles').insert([{userId:user1._id,channelId:channel1._id},{}], function(err, docs) {
var article1 = docs[0], article2 = docs[1];
}
});
};
You can see there are a lot of nested callbacks there, so I'm trying to use q to refactor it.
I hope the code will look like:
Q.fcall(step1)
.then(step2)
.then(step3)
.then(step4)
.then(function (value4) {
// Do something with value4
}, function (error) {
// Handle any error from step1 through step4
})
.end();
But I don't know how to do it.
You'll want to use Q.nfcall, documented in the README and the Wiki. All Mongoose methods are Node-style. I'll also use .spread instead of manually destructuring .then.
var mongoose = require('mongoose');
mongoose.connect('mongo://localhost/test');
var conn = mongoose.connection;
var users = conn.collection('users');
var channels = conn.collection('channels');
var articles = conn.collection('articles');
function getInsertedArticles() {
return Q.nfcall(users.insert.bind(users), [{/*user1*/},{/*user2*/}]).spread(function (user1, user2) {
return Q.nfcall(channels.insert.bind(channels), [{userId:user1._id},{userId:user2._id}]).spread(function (channel1, channel2) {
return Q.nfcall(articles.insert.bind(articles), [{userId:user1._id,channelId:channel1._id},{}]);
});
})
}
getInsertedArticles()
.spread(function (article1, article2) {
// you only get here if all three of the above steps succeeded
})
.fail(function (error) {
// you get here if any of the above three steps failed
}
);
In practice, you will rarely want to use .spread, since you usually are inserting an array that you don't know the size of. In that case the code can look more like this (here I also illustrate Q.nbind).
To compare with the original one is not quite fair, because your original has no error handling. A corrected Node-style version of the original would be like so:
var mongoose = require('mongoose');
mongoose.connect('mongo://localhost/test');
var conn = mongoose.connection;
function getInsertedArticles(cb) {
// insert users
conn.collection('users').insert([{/*user1*/},{/*user2*/}], function(err, docs) {
if (err) {
cb(err);
return;
}
var user1 = docs[0], user2 = docs[1];
// insert channels
conn.collection('channels').insert([{userId:user1._id},{userId:user2._id}], function(err, docs) {
if (err) {
cb(err);
return;
}
var channel1 = docs[0], channel2 = docs[1];
// insert articles
conn.collection('articles').insert([{userId:user1._id,channelId:channel1._id},{}], function(err, docs) {
if (err) {
cb(err);
return;
}
var article1 = docs[0], article2 = docs[1];
cb(null, [article1, article2]);
}
});
};
}
getInsertedArticles(function (err, articles) {
if (err) {
// you get here if any of the three steps failed.
// `articles` is `undefined`.
} else {
// you get here if all three succeeded.
// `err` is null.
}
});
With alternative deferred promise implementation, you may do it as following:
var mongoose = require('mongoose');
mongoose.connect('mongo://localhost/test');
var conn = mongoose.connection;
// Setup 'pinsert', promise version of 'insert' method
var promisify = require('deferred').promisify
mongoose.Collection.prototype.pinsert = promisify(mongoose.Collection.prototype.insert);
var user1, user2;
// insert users
conn.collection('users').pinsert([{/*user1*/},{/*user2*/}])
// insert channels
.then(function (users) {
user1 = users[0]; user2 = users[1];
return conn.collection('channels').pinsert([{userId:user1._id},{userId:user2._id}]);
})
// insert articles
.match(function (channel1, channel2) {
return conn.collection('articles').pinsert([{userId:user1._id,channelId:channel1._id},{}]);
})
.done(function (articles) {
// Do something with articles
}, function (err) {
// Handle any error that might have occurred on the way
});
Considering Model.save instead of Collection.insert (quite the same in our case).
You don't need to use Q, you can wrap yourself the save method and return directly a Mongoose Promise.
First create an utility method to wrap the save function, that's not very clean but something like:
//Utility function (put it in a better place)
var saveInPromise = function (model) {
var promise = new mongoose.Promise();
model.save(function (err, result) {
promise.resolve(err, result);
});
return promise;
}
Then you can use it instead of save to chain your promises
var User = mongoose.model('User');
var Channel = mongoose.model('Channel');
var Article = mongoose.model('Article');
//Step 1
var user = new User({data: 'value'});
saveInPromise(user).then(function () {
//Step 2
var channel = new Channel({user: user.id})
return saveInPromise(channel);
}).then(function (channel) {
//Step 3
var article = new Article({channel: channel.id})
return saveInPromise(article);
}, function (err) {
//A single place to handle your errors
});
I guess that's the kind of simplicity we are looking for.. right? Of course the utility function can be implemented with better integration with Mongoose.
Let me know what you think about that.
By the way there is an issue about that exact problem in the Mongoose Github:
Add 'promise' return value to model save operation
I hope it's gonna be solved soon. I think it takes some times because they are thinking of switching from mpromise to Q: See here and then here.
Two years later, this question just popped up in my RSS client ...
Things have moved on somewhat since May 2012 and we might choose to solve this one in a different way now. More specifically, the Javascript community has become "reduce-aware" since the decision to include Array.prototype.reduce (and other Array methods) in ECMAScript5. Array.prototype.reduce was always (and still is) available as a polyfill but was little appreciated by many of us at that time. Those who were running ahead of the curve may demur on this point, of course.
The problem posed in the question appears to be formulaic, with rules as follows :
The objects in the array passed as the first param to conn.collection(table).insert() build as follows (where N corresponds to the object's index in an array):
[ {}, ... ]
[ {userId:userN._id}, ... ]
[ {userId:userN._id, channelId:channelN._id}, ... ]
table names (in order) are : users, channels, articles.
the corresopnding object properties are : user, channel, article (ie the table names without the pluralizing 's').
A general pattern from this article by Taoofcode) for making asynchronous call in series is :
function workMyCollection(arr) {
return arr.reduce(function(promise, item) {
return promise.then(function(result) {
return doSomethingAsyncWithResult(item, result);
});
}, q());
}
With quite light adaptation, this pattern can be made to orchestrate the required sequencing :
function cascadeInsert(tables, n) {
/*
/* tables: array of unpluralisd table names
/* n: number of users to insert.
/* returns promise of completion|error
*/
var ids = []; // this outer array is available to the inner functions (to be read and written to).
for(var i=0; i<n; i++) { ids.push({}); } //initialize the ids array with n plain objects.
return tables.reduce(function (promise, t) {
return promise.then(function (docs) {
for(var i=0; i<ids.length; i++) {
if(!docs[i]) throw (new Error(t + ": returned documents list does not match the request"));//or simply `continue;` to be error tolerant (if acceptable server-side).
ids[i][t+'Id'] = docs[i]._id; //progressively add properties to the `ids` objects
}
return insert(ids, t + 's');
});
}, Q());
}
Lastly, here's the promise-returning worker function, insert() :
function insert(ids, t) {
/*
/* ids: array of plain objects with properties as defined by the rules
/* t: table name.
/* returns promise of docs
*/
var dfrd = Q.defer();
conn.collection(t).insert(ids, function(err, docs) {
(err) ? dfrd.reject(err) : dfrd.resolve(docs);
});
return dfrd.promise;
}
Thus, you can specify as parameters passed to cascadeInsert, the actual table/property names and the number of users to insert.
cascadeInsert( ['user', 'channel', 'article'], 2 ).then(function () {
// you get here if everything was successful
}).catch(function (err) {
// you get here if anything failed
});
This works nicely because the tables in the question all have regular plurals (user => users, channel => channels). If any of them was irregular (eg stimulus => stimuli, child => children), then we would need to rethink - (and probably implement a lookup hash). In any case, the adaptation would be fairly trivial.
Today we have mongoose-q as well. A plugin to mongoose that gives you stuff like execQ and saveQ which return Q promises.