We Keep Coding

Better performance when saving large JSON file to MySQL

I have an issue.
So, my story is:
I have a 30 GB big file (JSON) of all reddit posts in a specific timeframe.
I will not insert all values of each post into the table.
I have followed this series, and he coded what I'm trying to do in Python.
I tried to follow along (in NodeJS), but when I'm testing it, it's way too slow. It inserts one row every 5 seconds. And there 500000+ reddit posts and that would literally take years.
So here's an example of what I'm doing in.
var readStream = fs.createReadStream(location)
oboe(readStream)
.done(async function(post) {
let { parent_id, body, created_utc, score, subreddit } = data;
let comment_id = data.name;
// Checks if there is a comment with the comment id of this post's parent id in the table
getParent(parent_id, function(parent_data) {
// Checks if there is a comment with the same parent id, and then checks which one has higher score
getExistingCommentScore(parent_id, function(existingScore) {
// other code above but it isn't relevant for my question
// this function adds the query I made to a table
addToTransaction()
})
})
})
Basically what that does, is to start a read stream and then pass it on to a module called oboe.
I then get JSON in return.
Then, it checks if there is a parent saved already in the database, and then checks if there is an existing comment with the same parent id.
I need to use both functions in order to get the data that I need (only getting the "best" comment)
This is somewhat how addToTransaction looks like:
function addToTransaction(query) {
// adds the query to a table, then checks if the length of that table is 1000 or more
if (length >= 1000) {
connection.beginTransaction(function(err) {
if (err) throw new Error(err);
for (var n=0; n<transactions.length;n++) {
let thisQuery = transactions[n];
connection.query(thisQuery, function(err) {
if (err) throw new Error(err);
})
}
connection.commit();
})
}
}
What addToTransaction does, is to get the queries I made and them push them to a table, then check the length of that table and then create a new transaction, execute all those queries in a for loop, then comitting (to save).
Problem is, it's so slow that the callback function I made doesn't even get called.
My question (finally) is, is there any way I could improve the performance?
(If you're wondering why I am doing this, it is because I'm trying to create a chatbot)
I know I've posted a lot, but I tried to give you as much information as I could so you could have a better chance to help me. I appreciate any answers, and I will answer the questions you have.

browser functions not working inside callback function in protractor

I have been working on protractor for over two weeks now and have taken a liking towards it, but there's a certain problem that's been bugging me for a while:
What I want to do is: fetch values from database and use it in my tests, so I have been doing the following:
var test=function(callback)
{
//connection made;
//select query fired;
connection.query('select * from mytable', function(rows,error){
var result={};
for(int i=0;i<rows.length;i++)
{
result.first=rows[i].col1;
result.sec=rows[i].col2;
result.third=rows[i].col3;
callback(result);
}
});
}
//and then I use the return values as:
test(function(data){
var value1;
var value2;
value1=data.first;
value2=data.sec;
// i am able to print value1 and value2, the real problem is below:
browser.get(value1);//the browser function wont work!!
});
the spec passes with no failures,
I can print the values from the database but I am not able to use the browser functions.
Kindly help!!

How to add Javascript(jQuery) to a page created from XSLT/XML

I have an XML file (consisting of a few data items, namely <Book> ) and a corresponding XSLT file that I created, which when opened up in a browser turns this list of books into an html table. The columns are named "Author", "Title", and "BookID" (they are ids of child nodes of <tr>).
Now I want to make the resulting page dynamic using jQuery, i.e. I want to make the resulting table rows to be sorted on the column I click on. However, while the table renders fine, the resulting jQuery code seems to have no effect.
I am not sure whether this is a result of bugs in my jQuery code, or whether I didn't include it properly, or both. I included two <script></script> tags in my XSL file (one to hotlink the jQuery library, the other to link to my code), but I'm not sure if this is the correct way to do it. also, can someone look over my jQuery code to find out if there's anything wrong (I'm a complete newbie to web programming, so please forgive my errors)?
Thanks!
$(document).ready(function() {
function sortBy(a, b, selectFunction) {
var a1 = selectFunction(a);
var b1 = selectFunction(b);
if (a1 < b1) {
return -1;
}
if (a1 > b1) {
return 1;
}
return 0;
}
function sortHelper(index) {
var rows = $('table tbody tr').get();
rows.sort(function(a, b) {
return sortBy(a, b, function(x) { return $(x).children('td').eq(index).text().toUpperCase(); });
});
rows.appendTo('tbody');
}
$('#Author').click(function() {
sortHelper(0);
});
$('#Title').click(function() {
sortHelper(1);
});
$('#BookID').click(function() {
sortHelper(2);
});
});

A stated in the comments .get() returns a javascript object. So to use rows.sort() you want the jQuery object.
// javascript
$(obj).get(0); // returns the first element from the query
// jquery
$(obj).eq(0); // return the first $(element) from the query.
Also one thing I noticed:
since you're accessing the td's by an id you can do something like:
var topRow = $("table tbody tr").eq(0),
topCells = topRow.find("td"); // expecting #author, #title, #bookid
topCells.click(function(){
sortHelper($(this).index()); // makes more sense this way
});
Other than that if you're loading external *.js files into your solution you'll be fine. If you're inserting code directly into the page use CDATA encoding as described here.

How to make a clean Asynchronous loop?

Following typical REST standards, I broke up my resources into separate endpoints and calls. The main two objects in question here are List and Item (and of course, a list has a list of items, as well as some other data associated with it).
So if a user wants to retrieve his lists, he might make a Get request to api/Lists
Then the user might want to get the items in one of those lists and make a Get to api/ListItems/4 where 4 was found from List.listId retrieved in the previous call.
This is all well and good: the options.complete attribute of $.ajax lets me point to a callback method, so I can streamline these two events.
But things get very messy if I want to get the elements for all the lists in question. For example, let's assume I have a library function called makeGetRequest that takes in the end point and callback function, to make this code cleaner. Simply retrieving 3 elements the naive way results in this:
var success1 = function(elements){
var success2 = function(elements){
makeGetRequest("api/ListItems/3", finalSuccess);
}
makeGetRequest("api/ListItems/2", success2);
}
makeGetRequest("api/ListItems/1", success1);
Disgusting! This is the kind of thing in programming 101 we're smacked across the wrists for and pointed to loops. But how can you do this with a loop, without having to rely on external storage?
for(var i : values){
makeGetRequest("api/ListItems/" + i, successFunction);
}
function successFunction(items){
//I am called i-many times, each time only having ONE list's worth of items!
}
And even with storage, I would have to know when all have finished and retrieved their data, and call some master function that retrieves all the collected data and does something with it.
Is there a practice for handling this? This must have been solved many times before...

Try using a stack of endpoint parameters:
var params = [];
var results [];
params.push({endpoint: "api/ListItems/1"});
params.push({endpoint: "api/ListItems/2"});
params.push({endpoint: "api/ListItems/3"});
params.push({endpoint: "api/ListItems/4"});
Then you can make it recursive in your success handler:
function getResources(endPoint) {
var options = {} // Ajax Options
options.success = function (data) {
if (params.length > 0) {
results.push({endpoint: endpoint, data: data});
getResources(params.shift().endpoint);
}
else {
theMasterFunction(results)
}
}
$.get(endPoint, options)
}
And you can start it with a single call like this:
getResources(params.shift().endpoint);
Edit:
To keep everything self contained and out of global scope you can use a function and provide a callback:
function downloadResources(callback) {
var endpoints = [];
var results [];
endpoints.push({endpoint: "api/ListItems/1"});
endpoints.push({endpoint: "api/ListItems/2"});
endpoints.push({endpoint: "api/ListItems/3"});
endpoints.push({endpoint: "api/ListItems/4"});
function getResources(endPoint) {
var options = {} // Ajax Options
options.success = function (data) {
if (endpoints.length > 0) {
results.push({endpoint: endpoint, data: data});
getResources(endpoints.shift().endpoint);
}
else {
callback(results)
}
}
$.get(endPoint, options)
}
getResources(endpoints.shift().endpoint);
}
In use:
downloadResources(function(data) {
// Do stuff with your data set
});

dmck's answer is probably your best bet. However, another option is to do a bulk list option, so that your api supports requests like api/ListItems/?id=1&id=2&id=3.
You could also do an api search endpoint, if that fits your personal aesthetic more.

Asynchronous Javascript

function buildRpt() {
/* Initialize document here */
db.transaction(
function(trn) {
trn.executeSql(
'SELECT * FROM table',
null,
function(trn,result) {
for (var iI = 0; iI < result.rows.length; iI++) {
var row = result.rows.item(iI);
/* Add record data to document here */
trn.executeSql(
'SELECT * FROM detail table WHERE id = ?',
[ row.id ],
function (trn,rslt) {
/* Add detail info to document */
for (var iTmp = 0; iTmp < rslt.rows.length; iTmp++) {
var tmpRow = rslt.rows.item(iTmp);
/* update document */
}
},
errorHandler
);
}
},
errorHandler
);
}
);
}
I need to get information from the client side database and use it to populate a document. I can iterate through the records of a table and I'm fine. However, when I try to do another database query to get detail information for each record, javascript does the inner query asynchronously. I looked at other questions but I am still kind of foggy on how to accomplish this task. In the code above, the nested executeSql is processed asynchronously. The detail information never ends up in the document. Is there a way to get the detail information for each record into the document where it belongs?

I'm not sure what the question is? If the problem is that nothing comes back use console.log to see if data is coming out. The issue is likely due to your reference to the DOM that is being updated. If the issue is getting requests in a particular order read below.
Deferred objects
It would be nice to see if you have a framework available to you. Since you are issuing AJAX requests in a loop, you need something to synchronize those requests as you will not know in what order they will be returned. For example,
requestRows ->
rows.each ->
ajax call -> //These return in any order
If you insist on doing this framework free look into deferred objects, and be prepared to do a lot of work. Otherwise if you use dojo use http://dojotoolkit.org/reference-guide/dojo/Deferred.html or jQuery I have written something that does this: https://gist.github.com/1219564