I'm working on creating a JavaScript file to get a JSON dump of an entire MySQL database, running on server side. I found and am using the MySQL driver for node.js (https://www.npmjs.com/package/mysql) for queries, it's been straight forward enough to start. My issue is that I need to call multiple queries and get the results from all of them to put into a single JSON file and I can't quite get that to work. I'm entirely new to JavaScript (basically never touched it before now) so it's probably a relatively simple solution that I'm just missing.
Currently I do a query of 'SHOW TABLES' to get a list of all the tables (this can change so I can't just assume a constant list). I then just want to basically loop through the list and call 'SELECT * from table_name' for each table, combining the results as I go to get one big JSON. Unfortunately I haven't figured out how to get the code to finish all the queries before trying to combine them, thus retuning 'undefined' for all the results. Here is what I currently have:
var mysql = require('mysql');
var fs = require('fs');
var connection = mysql.createConnection({
host: 'localhost',
user: 'root',
password: 'pass',
database: 'test_data'
});
connection.connect();
connection.query('SHOW TABLES;', function(err, results, fields)
{
if(err) throw err;
var name = fields[0].name;
var database_json = get_table(results[0][name]);
for (i = 1; i < results.length; i++)
{
var table_name = results[i][name];
var table_json = get_table(table_name);
database_json = database_table_json.concat(table_json);
}
fs.writeFile('test_data.json', JSON.stringify(database_json), function (err)
{
if (err) throw err;
});
connection.end();
});
function get_table(table_name)
{
connection.query('select * from ' + table_name + ';', function(err, results, fields) {
if(err) throw err;
return results;
});
}
This gets the table list and goes through all of it with no issue, and the information returned by the second query is correct if I just do a console.log(results) inside the query, but the for loop just keeps going before any query is completed and thus 'table_json' just ends up being 'undefined'. I really think this must be an easy solution (probably something with callbacks which I don't quite understand fully yet) but I keep stumbling.
Thanks for the help.
I'm guessing that this is for some sort of maintenance type function and not a piece that you need for your application. You're probably safe to do this asynchronously. This module is available here: https://github.com/caolan/async
You can also use Q promises, available here: https://github.com/kriskowal/q
This answer: describes both approaches pretty well: Simplest way to wait some asynchronous tasks complete, in Javascript?
Related
I am still new to node.js so I apologize in advance if this question is silly.
First I have tried to execute the test.js with command prompt
conn.js
var mysql = require('mysql');
var connection = mysql.createConnection({
host: 'localhost',
user: 'root',
password: '',
database: 'tgs'
});
connection.connect(function(err) {
if (err) throw err;
});
module.exports = connection;
test.js
function testconnect()
{
var conn = require('./conn');
var sql = "INSERT INTO chat_line (line_text) VALUES ('test')";
conn.query(sql, function (err, result) {
if (err) throw err;
});
}
testconnect();
in the command prompt
D:\xampp\htdocs\TGS\>node test.js
and it works, I did get the input in my database. After that I wanted to try to implement this in my system so I have tried to execute this in php
testing.php
<script>
function testalert()
{
alert("alert1");
var conn = require('./conn');
var sql = "INSERT INTO chat_line (line_text) VALUES ('test')";
conn.query(sql, function (err, result) {
if (err) throw err;
});
alert("alert2");
}
</script>
<input type="button" onclick="testalert();">
and it doesn't work. The first alert did pop up after clicking the button but the second alert did not, also there is no input in my database.
Node.js and PHP are different technologies, you cannot directly use one in another if you want to interact with one another you will need some mediator like (API, queue, database etc).
try to understand how nodeJs works, there is a dependency of mysql library at line one, which is a middleware helping with dealing with database
var mysql = require('mysql');
above line can only work in nodeJs environment, if you want to use it in PHP then there will be some other library in PHP to do same (I am not PHP guy so cannot help with ideal one here) and you will not need nodeJS at all with your example.
do let me know if I made it clear.
I loop over some directories and then the files in them. I process the files by directory and then try to add the processed results into MySQL.
I call conn.query('INSERT QUERY HERE') and it seems to continue on but the query never runs on the server. If I tell it to just process one directory and wait till the end it will run the queries but I can't have it continue to store all the queries in memory till the end of the script or node will fail out due to mem cap. I have tried everything I can think of to try and force the queued queries to run but no luck.
Here is an example of my code
dirs.forEach(function(dir){
var data = [];
var connection = mysql.createConnection(conConfig);
files.forEach(function(file){
//do some processing on files push into data array
//creating array of objects
});
data.forEach(function(record){
connection.query('INSERT INTO TABLE SET ?', record);
});
connection.end();
});
The code will just continue to loop over the directories without ever sending the query to mysql. I know it will work by limiting the code to just run on one directory and it will runt he queries once the one directory is processed but not if I let it run on all directories.
I have tried using mysql pooling as well with no luck. The
pool.on('enqueue' function... will fire but never send it over to the server.
edit:
So I tried calling the script with a for loop from bash to call every dir name individually and all records were loaded. I'm dumbfounded as to why a mysql connection is never established in my orig example.
Javascript calls the mysql query asynchronously. That means the connection will likely be be closed before all insert queries are finished.
What you can do is to use the callbacks that the query function provides:
var qCnt = array.length;
var connection = mysql.createConnection(conConfig);
connection.connect();
array.forEach(function(record){
connection.query('INSERT INTO TABLE SET ?', record, function(err){
if (err) throw err;
qCnt--;
if (qCnt == 0){
connection.end();
}
});
});
This solution is not ideal, since all the insert queries are fired regardless of your database connection limit etc. you may want to fire the next insert only after the former is done. This is also possible with some tricks.
It is in fact an async issue. There does not seem to be any way to force queued queries to execute without stopping the current running process. I had to use the async module in order to make my code work.
#luksch connection.end() will not close the connection till all queued queries are finished. I did use his iteration method to make the callback though.
Here is how I did it.
var async = require('async');
var connection = mysql.createConnection(conConfig);
var dirs = fs.readdirSync('./rootdirectory');
async.eachSeries(dirs,function(dir,callback){
var data = [];
files.forEach(function(file){
//do some processing on files push into data array
//creating array of objects
});
var qCount = data.length;
data.forEach(function(record){
connection.query('INSERT INTO TABLE SET ?', record, function(err){
if (err) throw err;
qCount--
if(qCount === 0) { callback(true); }
});
});
function(){ connection.end(); }
});
This will iterate over the directories and queue all the queries and then force the queries to be run till all directories have been processed then call the final function to close the connection.
I am querying a mySQL database from the context of Node.js server code. Generally speaking, things are working fine. However, I can't figure out a way to tell whether an UPDATE statement does not update any rows. For instance, if I have code such as:
connection.query("UPDATE table SET columnA = valA WHERE columnB = valB;",
function(err, rows, fields) {
..some code..
}
As far as I can tell, none of the callbacks (err, rows, fields) will provide me with any information informing me that zero rows were affected. This is important information to me. How can I resolve this? Thanks!
I assume you're using node-mysql. The count of changed rows is in result.changedRows variable.
Try:
connection.query('UPDATE table SET ...', function (err, result) {
if (err) throw err;
console.log('changed ' + result.changedRows + ' rows');
});
You can also get result.affectedRows.
Further info is within the docs.
I have the following code. The idea is that I update a database row in an interval, however if I remove the row manually from the database while this script runs, the save() still goes into success(), but the row is not actually put back into the database. (Because sequelize does an update query with a where clause and no rows match.) I expected a new row to be created or error() to be called. Any ideas to what I can do to make this behave like I want to?
var Sequelize = require("sequelize")
, sequelize = new Sequelize('test', 'test', 'test', {logging: false, host: 'localhost'})
, Server = sequelize.import(__dirname + "/models/Servers")
sequelize.sync({force: true}).on('success', function() {
Server
.create({ hostname: 'Sequelize Server 1', ip: '127.0.0.1', port: 0})
.on('success', function(server) {
console.log('Server added to db, going to interval');
setInterval(function() { console.log('timeout reached'); server.port = server.port + 1; server.save().success(function() { console.log('saved ' + server.port) }).error(function(error) { console.log(error); }); }, 1000);
})
})
I'm afraid what you are trying to do is not currently supported by sequelize.
Error callbacks are only ment for actual error situations, i.e. SQL syntax errors, stuff like that. Trying to update a non-existing row is not an error in SQL.
The import distinction here is, that you are modifying your database outside of your program. Sequelize has no way of knowing that! I have two possible solutions, only one of which is viable right now:
1 (works right now)
Use sequelize.query to include error handling in your query
IF EXISTS (SELELCT * FROM table WHERE id = 42)
UPDATE table SET port = newport WHERE id = 42
ELSE
INSERT INTO table ... port = newport
Alternatively you could create a feature request on the sequelize github for INSERT ... ON DUPLICATE KEY UPDATE syntax to be implemented see and here
2 (will work when transactions are implemented
Use transactions to first check if the row exists, and insert it if it does not. Transactions are on the roadmap for sequelize, but not currently supported. If you are NOT using connection pooling, you might be able to acomplish transactions manually by calling sequelize.query('BEGIN / COMMIT TRANSACTION').
I am using node.js and with the native mongodb driver (node-mongodb-native);
My current project uses node.js + now.js + mongo-db.
The system basically sends data from the browser to node.js, which is processed with haskell and later fed back to the browser again.
Via a form and node.js the text is inserted in a mongo-db collection called "messages".
A haskell thread reads the entry and stores the result in the db collection "results". This works fine.
But now I need the javascript code that waits for the result to appear in the collection results.
Pseudo code:
wait until the collection result is non-empty.
findOne() from the collection results.
delete the collection results.
I currently connect to the mongodb like this:
var mongo = require('mongodb'),
Server = mongo.Server,
Db = mongo.Db;
var server = new Server('localhost', 27017, {
auto_reconnect: true
});
var db = new Db('test', server);
My haskell knowledge is quite good but not my javascript skills.
So I did extensive searches, but I didn't get far.
glad you solved it, i was going to write something similar:
setTimeout(function(){
db.collection('results',function(coll){
coll.findOne({}, function(err, one){
if( err ) return callback(err);
coll.drop(callback); //or destroy, not really sure <-- this will drop the whole collection
});
});
} ,1000);
The solution is to use the async library.
var async = require('async');
globalCount = -1;
async.whilst(
function () {
return globalCount<1;
},
function (callback) {
console.log("inner while loop");
setTimeout(db_count(callback), 1000);
},
function (err) {
console.log(" || whilst loop finished!!");
}
);