We Keep Coding

Better performance when saving large JSON file to MySQL

I have an issue.
So, my story is:
I have a 30 GB big file (JSON) of all reddit posts in a specific timeframe.
I will not insert all values of each post into the table.
I have followed this series, and he coded what I'm trying to do in Python.
I tried to follow along (in NodeJS), but when I'm testing it, it's way too slow. It inserts one row every 5 seconds. And there 500000+ reddit posts and that would literally take years.
So here's an example of what I'm doing in.
var readStream = fs.createReadStream(location)
oboe(readStream)
.done(async function(post) {
let { parent_id, body, created_utc, score, subreddit } = data;
let comment_id = data.name;
// Checks if there is a comment with the comment id of this post's parent id in the table
getParent(parent_id, function(parent_data) {
// Checks if there is a comment with the same parent id, and then checks which one has higher score
getExistingCommentScore(parent_id, function(existingScore) {
// other code above but it isn't relevant for my question
// this function adds the query I made to a table
addToTransaction()
})
})
})
Basically what that does, is to start a read stream and then pass it on to a module called oboe.
I then get JSON in return.
Then, it checks if there is a parent saved already in the database, and then checks if there is an existing comment with the same parent id.
I need to use both functions in order to get the data that I need (only getting the "best" comment)
This is somewhat how addToTransaction looks like:
function addToTransaction(query) {
// adds the query to a table, then checks if the length of that table is 1000 or more
if (length >= 1000) {
connection.beginTransaction(function(err) {
if (err) throw new Error(err);
for (var n=0; n<transactions.length;n++) {
let thisQuery = transactions[n];
connection.query(thisQuery, function(err) {
if (err) throw new Error(err);
})
}
connection.commit();
})
}
}
What addToTransaction does, is to get the queries I made and them push them to a table, then check the length of that table and then create a new transaction, execute all those queries in a for loop, then comitting (to save).
Problem is, it's so slow that the callback function I made doesn't even get called.
My question (finally) is, is there any way I could improve the performance?
(If you're wondering why I am doing this, it is because I'm trying to create a chatbot)
I know I've posted a lot, but I tried to give you as much information as I could so you could have a better chance to help me. I appreciate any answers, and I will answer the questions you have.

Copy a Parse.com class to new class with transformation of values

There is an existing Parse.com class that needs to be copied to a new Parse.com class with some new columns and the transformation of one of the columns. The code currently works and uses the Parse.Query.each method to iterate over all records as detailed in the Parse.com documentation but it stops processing at 831 records although there are 12k+ records in the class. This is odd given each should not have a limit and other default limits are 100 or 1000 for find. Should another method be used to iterate over all records or is there something wrong with the code?
var SourceObject = Parse.Object.extend("Log_Old_Class");
var source_query = new Parse.Query(SourceObject);
var TargetObject = Parse.Object.extend("Log_New_Class")
source_query.each(function(record) {
//save record to new class code works fine
var target_query = new TargetObject();
target_query.set("col1_new",record.col1);
target_query.set("col2_new",record.col2);
//etc...
target_query.save(null, {
success: function(obj) {
//SAVED
},
error: function(obj, error) {
//ERROR
}
});
}).then(function() {
//DONE
},
function(error) {
//error
});

One thing that comes to my mind immediately is that the function is getting timed-out. Parse has time limitations on each function. If I were you, I'd first load all the objects in the source class and then add them separately by having a delay between to API calls (server overload issues can also be present).

unexpected identifier - javascript,node and mongodb

Running the code below I get an unexpected identifier error when checking if the documents state has changed. I've Googled this to death. tried to find documentation at Mongodb, here, and other misc sources without any luck. Ive spent the whole day on this and feel pretty stupid by now.
All help is sincerely appreciated.
var MongoClient = require('mongodb').MongoClient;
MongoClient.connect('mongodb://localhost:27017/course', function(err, db) {
if (err) throw err;
var weather = db.collection('weather');
var filter = {};
var projection ={'State':1, 'Temperature':1, _id:true};
var options = { 'skip' : 0,
'limit' : 20000 ,
'sort' : [['State', 1], ['Temperature', -1]] };
var cursor = weather.find(filter, projection, options);
var month_high = false;
var prevState = '';
var curID = '';
var operation = {'_id':curID,'$set':{'month_high':true}};
// get first document
cursor.first();
// initialize vars
prevState = doc.State;
// cycle through documents
cursor.each(function(err, doc) {
if(err) throw err;
if(doc == null) {
return db.close();
}
//Save current document Id
curID = doc._id;
// Check if State has changgd
if prevState != doc.state{
//If State has changed update the document
weather.update(operation,function(err,doc));
console.dir(doc); //we expect four documents to ouput.
}
// save current doc.state as previous state
prevState = doc.State;
;})
});
});

I think an unexpected identifier would be difficult to find on google. You may have extra closing braces or closing parentheses.
I don't see a closing parentheses for this opening parentheses:
MongoClient.connect(

Change the if prevState != doc.state{ to if (prevState != doc.state) { -- that is: add a pair of ( ).
To easily find such problems, you should use a Javascript syntax checker. Google this term and you'll get some good hits.
Also note that you have at least one other syntax problem in your code in
weather.update(operation,function(err,doc)); you use the function keyword but does not provide a function body. To make it pass syntax checks you should at least do: weather.update(operation,function(err,doc) {});
but you may want to add some logic inside the curly braces {}.
This was detected by http://esprima.org/demo/validate.html
One more thing, unrelated to the original question:
you use throw in your code for reporting errors. As the node/mongo predominantly use async APIs throwing an exception is almost meaningless. Take a look at questions such as:
Node.js Best Practice Exception Handling
Nested callbacks and exceptions handling in NodeJS
Catching exceptions from callbacks

Get specific value from Firebase JSON tree based on value

I am writing a Javascript web app using Firebase. If I have data like this in my JSON tree:
users
session1
screenname:Bill
session2
screenname:Steve
...and I know the session number, how can I grab the "name" of the user? I have tried this:
valUsers.child('session1').child('screenname').once('value', function(data){
console.log(data);
});
But that does not seem to work. I'm guessing I have a syntax issue?

You were pretty close, but needed to use DataSnapshot.val() as shown below to access the point you want.
var ref = new Firebase('yourfirebaselocation/users');
var sessionNum = 'session1';
ref.child(sessionNum + '/screenname').once('value', function(dataSnapshot) {
var specificScreenname = specificPoint.val();
}
When working with Firebase, you may want to also explore working with Promise objects. This would allow you to create some repeatable functions that can reuse your reference on multiple locations and also give Firebase time to respond to your query.
var ref = new Firebase('yourfirebaselocation/users');
function getUser(theRef, location) {
return new Promise(
function(resolve, reject) {
theRef.child(location).once('value', function(dataSnapshot) {
resolve(dataSnapshot);
});
});
}
getUser(ref, specificlocationvariable).then( function(val) {
specificScreenname = val.val();
});
You don't specify other than 'JavaScript' so do keep in mind that Promises are an ECMAScript 6 proposal with some compatibility issues, but a lot of the JS frameworks have an implementation for it that smooth that issue (Angular, Ember, etc).

node.js/javascript/couchdb view to associative array does not seem to work

I am trying to create a webapp on a node/couchdb/windows stack but get terribly stung by what seems to be a lack of experience.
In the database, there is a view that returns all users with passwords. Based on the tutorial for a blog I have tried to access the view through my node code.
Whenever I investigate the structure of the users or users variable, I get an undefined object.
The call to getDatabase() has been tested elsewhere and works at least for creating new documents.
function GetUser(login)
{
var users = GetUsers();
return users[login];
}
function GetUsers() {
var db = getDatabase();
var usersByEmail = [];
db.view("accounts", "password_by_email")
.then(function (resp) {
resp.rows.forEach(function (x) { usersByEmail[x.key] = x.value});
});
//usersByEmail['test'] = 'test';
return usersByEmail;
}
I am aware that both the use of non-hashed passwords as well as reading all users from the database is prohibitive in the final product - just in case anyone wanted to comment on that.
In case something is wrong with the way I access the view: I am using a design document called '_design/accounts' with the view name 'password_by_email'.

Your call to db.view is asynchronous, so when you hit return usersByEmail the object hasn't yet been populated. You simply can't return values from async code; you need to have it make a callback that will execute the code that relies on the result.