Big json(like 100MB) to excel in javascript [closed] - javascript

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
I'am new to REST technology. I'm currently dealing with a json response from a server and presenting the data on the client side.
I am now getting around 22MB of json data, and I need to export it to an excel sheet.
My problem:
When I'm iterating to json , the browser is not responding. What am I doing wrong?

IDEALLY, you should not be exporting such a big excel file for download. You must be breaking the data to chunks and asking user to download in batches. However, if it may be a part of your requirement, to handle it, refer answer below.
You might have to download the json in batches and combine as a single excel file and ask user to download!
The unresponsive script dialog box shows when some javascript thread takes too long too complete. Editing the registry could work, but you would have to do it on all client machines. You could use a "recursive closure" as follows to alleviate the problem. It's just a coding structure in which allows you to take a long running for loop and change it into something that does some work, and keeps track where it left off, yielding to the browser, then continuing where it left off until we are done.
Figure 1, Add this Utility Class RepeatingOperation to your javascript file. You will not need to change this code:
RepeatingOperation = function(op, yieldEveryIteration) {
//keeps count of how many times we have run heavytask()
//before we need to temporally check back with the browser.
var count = 0;
this.step = function() {
//Each time we run heavytask(), increment the count. When count
//is bigger than the yieldEveryIteration limit, pass control back
//to browser and instruct the browser to immediately call op() so
//we can pick up where we left off. Repeat until we are done.
if (++count >= yieldEveryIteration) {
count = 0;
//pass control back to the browser, and in 1 millisecond,
//have the browser call the op() function.
setTimeout(function() { op(); }, 1, [])
//The following return statement halts this thread, it gives
//the browser a sigh of relief, your long-running javascript
//loop has ended (even though technically we havn't yet).
//The browser decides there is no need to alarm the user of
//an unresponsive javascript process.
return;
}
op();
};
};
Figure 2, The following code represents your code that is causing the 'stop running this script' dialog because it takes so long to complete:
process10000HeavyTasks = function() {
var len = 10000;
for (var i = len - 1; i >= 0; i--) {
heavytask(); //heavytask() can be run about 20 times before
//an 'unresponsive script' dialog appears.
//If heavytask() is run more than 20 times in one
//javascript thread, the browser informs the user that
//an unresponsive script needs to be dealt with.
//This is where we need to terminate this long running
//thread, instruct the browser not to panic on an unresponsive
//script, and tell it to call us right back to pick up
//where we left off.
}
}
Figure 3. The following code is the fix for the problematic code in Figure 2. Notice the for loop is replaced with a recursive closure which passes control back to the browser every 10 iterations of heavytask()
process10000HeavyTasks = function() {
var global_i = 10000; //initialize your 'for loop stepper' (i) here.
var repeater = new this.RepeatingOperation(function() {
heavytask();
if (--global_i >= 0){ //Your for loop conditional goes here.
repeater.step(); //while we still have items to process,
//run the next iteration of the loop.
}
else {
alert("we are done"); //when this line runs, the for loop is complete.
}
}, 10); //10 means process 10 heavytask(), then
//yield back to the browser, and have the
//browser call us right back.
repeater.step(); //this command kicks off the recursive closure.
};
Adapted from this source:
http://www.picnet.com.au/blogs/Guido/post/2010/03/04/How-to-prevent-Stop-running-this-script-message-in-browsers

Related

Queuing users for Slides API Batch Update in App Script [duplicate]

This question already has answers here:
How to understand LockService and implement it correctly?
(2 answers)
Closed 1 year ago.
My users in App Script read their queue number from a Spreadsheet:
var array = sessionsSheets.getRange(row, 2, 1, 3).getValues()[0];
var questionNumber = array[2]; //This is the variable of interest, it is an integer signifying the queue
sessionsSheets.getRange(`D${row}`).setFormula(questionNumber+1); //This is the updated queue
They then update this queue number as seen above. This method works fine most of the time, but if you take two devices and run the script simultaneously, they both receive the same queue, and the script will halt for one of them because you need a unique queue number later on to use Slides API:
Slides.Presentations.batchUpdate({'requests': requests}, presentationId);
If promises worked, I would have simply put the Slides API line in a try block, and if an error of duplicity pops up, I would then call the same function recursively, up until overlap doesn't occur. However, promises are not supported in App Script, so what should I try instead?
I found a fix for my special case:
I used ScriptProperties in the Properties Services in App Script. When a user opens the application, he looks for the property of the key with his session's token. If it says busy, he waits and recursively tries the same function, if it says free, he changes it to busy, executes his APIs, then at the end changes it to free again.
Code Snippet:
function queueing(comment,name,token,sessionsSheets,row) {
var properties = PropertiesService.getScriptProperties();
var state = properties.getProperty(token);
if (state==null){
properties.setProperty(token , `busy#0`);
posting(comment,name,token,sessionsSheets,row,0);
} else if (state.includes("busy")){
Logger.log('Gotta wait my turn');
Utilities.sleep(2000);
queueing(comment,name,token,sessionsSheets,row);
} else if (state.includes("free")){
var questionNumber = state.split('#')[1];
properties.setProperty(token , `busy#${questionNumber}`);
posting(comment,name,token,sessionsSheets,row,questionNumber);
}}

NodeJs/expressjs : Run lengthy code in a callback [duplicate]

This question already has answers here:
Long-running computations in node.js
(3 answers)
Closed 8 years ago.
Callbacks are asynchronous , So does that mean that if I run a lengthy computation in a callback it wont affect my main thread ?
For example:
function compute(req,res){ // this is called in an expressjs route.
db.collection.find({'key':aString}).toArray(function(err, items) {
for(var i=0;i<items.length;i++){ // items length may be in thousands.
// Heavy/lengthy computation here, Which may take 5 seconds.
}
res.send("Done");
});
}
So, the call to database is ascnchronous. Does that mean the for loop inside the callback will NOT block the main thread ?
And if it is blocking, How may I perform such things in an async way?
For the most part, node.js runs in a single thread. However, node.js allows you to make calls that execute low-level operations (file reads, network requests, etc.) which are handled by separate threads. As such, your database call most likely happens on a separate thread. But, when the database call returns, we return back to the main thread and your code will run in the main thread (blocking it).
The way to get around this is to spin up a new thread. You can use cluster to do this. See:
http://nodejs.org/api/cluster.html
Your main program will make the database call
When the database call finishes, it will call fork() and spin up a new thread that runs your-calculations.js and sends an event to it with any input data
your-calculations.js will listen for an event and do the necessary processing when it handles the event
your-calculations.js will then send an event back to the main thread when it has finished processing (it can send any output data back)
If the main thread needs the output data, it can listen for the event that your-calculations.js emits
If you can't do, or don't want to use a thread, you can split up the long computation with setImmediates. e.g. (writing quickly on my tablet so may be sloppy)
function compute(startIndex, max, array, partialResult, callback) {
var done = false;
var err = null;
var stop = startIndex+100; // or some reasonable amount of calcs...
if (stop >= max) {
stop = max;
done = true;
}
// do calc from startIndex to stop, using partialResult as input
if (done)
callback(err, result);
else
process.setImmediate ( go look this part up or I'll edit tomorrow)...
But the idea is you call youself again with start += 100.
}
In between every 100 calculations node will have time to process other requests, handle other callbacks, etc. Of course, if they trigger another huge calculation evedntually things will grind to a halt.

Display message before script block

I have a hefty script to run synchronously and I want to display a message like "Processing, please wait..." before it runs so the user isn't wondering why the page is frozen for a few seconds. I'm trying to do something like:
messageBox.html("Processing, please wait...");
// run hefty script
messageBox.html("Finished!");
But the page blocks before the message is displayed, even though the messageBox.html() statement comes first. Why is this?
Sometimes it makes sense to fire the "hefty script" in a timeout.
messageBox.html("Processing, please wait...");
setTimeout(function () {
heftyScript();
messageBox.html("Finished!");
}, 1);
The reason this happens is because it often holds UI updates until the end of the event loop (after your "hefty" script has finished). Setting a timeout ensures that hefty script doesn't run until a subsequent iteration of the event loop (letting the UI update at the end of the current iteration beforehand).
I would consider Web Workers in this case: http://www.html5rocks.com/en/tutorials/workers/basics/
UPDATE:
If for some reason you cannot use them then you should split your processing on smaller chunks and run them asynchronously. Locking UI is not an option.
Here is what you can do:
function heftyScript() {
var arr = [...];
var chunk_start = 0;
function do_chunk() {
for( var i = 0; i < 100; ++i ) { // 100 items per chunk
if( chunk_start >= arr.length)
return;
process( arr[chunk_start++] ); // process one element
}
window.setTimeout(do_chunk,10);
}
do_chunk();
}
It depends where the time is being spent. If it's downloading and parsing the JavaScript file, the messageBox.html() must be pure html or you do it in a script block before referencing the external file. If the time spent is running that long function then setTimeout(function () { heftyScript(); messageBox.html('finished'); }, 1); works wonderfully.

how alert make javascript interrupt? can I do it with code?

I'm procesing a kind of "big" JSON object around of 4000 elements passing for different methods, and I would like to update a div tag with a text showing the process.
But for some strange reason (just tested in Firefox and Chrome), they don't update the DOM object with the text.
$("#estatusBar").text(_item.Centro_de_trabajo);
Both prefer to continue calculating all the data and other process without and dont take the time for update the text. But if I just code an Alert("") in the loop and then in chrome I click on the "selected box" saying ignore all other alerts, chrome suddenly starts updating the text.
So I was thinking if I can "pause" the calculation with some kind of code to stop and update the DOM element and then continue making the other process?
Is this possible or what is an alternative to this strange behavior?
-- EDIT --
this is the code of the loop
$.each(plantillas, function(_index, _item){
updateBar(_item.Centro_de_trabajo);
calculateItem(_item,_index);
a.push("<div class='blockee'><ul>"+ /*temp.join("")*/ t(_item) +"</ul></div>");
});
No you cannot do what alert does. This limitation is really annoying in some cases but if your problem is just a progress for a single long computation then the solution is simple.
Instead of doing alll the records in one single loop break the computation in "small enough" chunks and then do something like
function doit()
{
processBlockOfRecords();
updateProgressBar();
if (!finished()) {
setTimeout(doit, 0);
}
}
setTimeout(doit, 0);
With this approach is also simple to add an "abort" button to stop the computation.
In your example the loop is
$.each(plantillas, function(_index, _item){
updateBar(_item.Centro_de_trabajo);
calculateItem(_item,_index);
a.push("<div class='blockee'><ul>"+ /*temp.join("")*/ t(_item) +"</ul></div>");
});
so the computation could be split with (untested)
function processRecords(plantillas, completion_callback) {
var processed = 0;
var result = [];
function doit() {
// Process up to 20 records at a time
for (var i=0; i<20 && processed<plantillas.length; i++) {
calculateItem(plantillas[processed], processed);
result.push("<div class='blockee'><ul>" +
t(plantillas[processed]) +
"</ul></div>");
processed++;
}
// Progress bar update
updateProgress(processed, plantillas.length);
if (processed < plantillas.length) {
// Not finished, schedule another block
setTimeout(doit, 0);
} else {
// Processing complete... inform caller
if (completion_callback) completion_callback(result);
}
}
// Schedule computation start
setTimeout(doit, 0);
}
You can try using web workers, to defer the calculation to the background, and this should help you out. Web workers were designed to do this very thing (push large calculations to a background thread). However, this may not work in all browsers. Your main concern is IE, and only IE 10 supports it:
Do you want some version of wait or pause or sleep or similar in javascript is that it?
The only way is with
window.setTimeout()
You can pause any function with it.
Check this post too might help:
Sleep/Pause/Wait in Javascript

Can I interrupt javascript code and then continue on a keystroke?

I am porting an old game from C to Javascript. I have run into an issue with display code where I would like to have the main game code call display methods without having to worry about how those status messages are displayed.
In the original code, if the message is too long, the program just waits for the player to toggle through the messages with the spacebar and then continues. This doesn't work in javascript, because while I wait for an event, all of the other program code continues. I had thought to use a callback so that further code can execute when the player hits the designated key, but I can't see how that will be viable with a lot of calls to display.update(msg) scattered throughout the code.
Can I architect things differently so the event-based, asynchronous model works, or is there some other solution that would allow me to implement a more traditional event loop?
Am I making sense?
Example:
// this is what the original code does, but obviously doesn't work in Javascript
display = {
update : function(msg) {
// if msg is too long
// wait for user input
// ok, we've got input, continue
}
};
// this is more javascript-y...
display = {
update : function(msg, when_finished) {
// show part of the message
$(document).addEvent('keydown', function(e) {
// display the rest of the message
when_finished();
});
}
};
// but makes for amazingly nasty game code
do_something(param, function() {
// in case do_something calls display I have to
// provide a callback for everything afterwards
// this happens next, but what if do_the_next_thing needs to call display?
// I have to wait again
do_the_next_thing(param, function() {
// now I have to do this again, ad infinitum
}
}
The short answer is "no."
The longer answer is that, with "web workers" (part of HTML5), you may be able to do it, because it allows you to put the game logic on a separate thread, and use messaging to push keys from the user input into the game thread. However, you'd then need to use messaging the other way, too, to be able to actually display the output, which probably won't perform all that well.
Have a flag that you are waiting for user input.
var isWaiting = false;
and then check the value of that flag in do_something (obviously set it where necessary as well :) ).
if (isWaiting) return;
You might want to implement this higher up the call stack (what calls do_something()?), but this is the approach you need.

Categories