We Keep Coding

Javascript for-loop async File execution

In the next code, I want to process several files at the same time without wait to the end of each other. For this reason, I first read the files (array) and then the callback is called to process an element of this array instance.
I have found a problem into this javascript code, exactly in a async for-loop, where this process is executed as a sync code instead of async.
var array = ['string1','string2','string3','string4'];
function processArray (arrayString,callback){
//Read file Example.csv thought sync way
try{
var ifs = new InputFileStream('Example.csv','utf8');
table = ifs.read(0);
ifs.close();
}catch(err){
console.log(err.stack);
}
callback(arrayString, table);
}
//Async for
for (var i=0; i<array.length; i++) {
processArray(array[i], function(arrayString, table){
//Here process the file values thought async way
console.log('processed_'+i);
});
}

You could put the call back in a setTimeout with a delay of 1ms. That will run it in the next block of execution and your loop will continue on.
e.g. use this:
setTimeout(function() { callback(arrayString, table); }, 1);
instead of this:
callback(arrayString, table);
An alternative to this is to run the callback on a separate thread using Web Workers. I don't think it would appropiate to provide a long answer describing how to do multi threaded JavaScript here so I'll just leave the link to the docs. https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers

where this process is executed as a sync code instead of async
I've seen that you just have find out the answers of your question, remember that JavaScript is single thread.
So, for that when you execute operations that require full use of CPU like for..loops, while, etc; you just will get your code running synchronous and not only that,
You will get your web page freeze if they are huge loops
Let me give you an example, this is a while loop that will run for 6 seconds, look how you cannot do anything in stackoverflow.
function blocker (ms) {
console.log('You cannot do anything')
var now = new Date().getTime();
while(true) {
if (new Date().getTime() > now +ms)
return;
}
}
blocker(6000) //This stop your entire web page for 6 seconds
If you really want to achieve running blocking code in the background read about Web Workers or you just can use a small library I wrote, that allow you to execute a blocking CPU function in the background, I called it GenericWebWorker

NodeJs/expressjs : Run lengthy code in a callback [duplicate]

This question already has answers here:
Long-running computations in node.js
(3 answers)
Closed 8 years ago.
Callbacks are asynchronous , So does that mean that if I run a lengthy computation in a callback it wont affect my main thread ?
For example:
function compute(req,res){ // this is called in an expressjs route.
db.collection.find({'key':aString}).toArray(function(err, items) {
for(var i=0;i<items.length;i++){ // items length may be in thousands.
// Heavy/lengthy computation here, Which may take 5 seconds.
}
res.send("Done");
});
}
So, the call to database is ascnchronous. Does that mean the for loop inside the callback will NOT block the main thread ?
And if it is blocking, How may I perform such things in an async way?

For the most part, node.js runs in a single thread. However, node.js allows you to make calls that execute low-level operations (file reads, network requests, etc.) which are handled by separate threads. As such, your database call most likely happens on a separate thread. But, when the database call returns, we return back to the main thread and your code will run in the main thread (blocking it).
The way to get around this is to spin up a new thread. You can use cluster to do this. See:
http://nodejs.org/api/cluster.html
Your main program will make the database call
When the database call finishes, it will call fork() and spin up a new thread that runs your-calculations.js and sends an event to it with any input data
your-calculations.js will listen for an event and do the necessary processing when it handles the event
your-calculations.js will then send an event back to the main thread when it has finished processing (it can send any output data back)
If the main thread needs the output data, it can listen for the event that your-calculations.js emits

If you can't do, or don't want to use a thread, you can split up the long computation with setImmediates. e.g. (writing quickly on my tablet so may be sloppy)
function compute(startIndex, max, array, partialResult, callback) {
var done = false;
var err = null;
var stop = startIndex+100; // or some reasonable amount of calcs...
if (stop >= max) {
stop = max;
done = true;
}
// do calc from startIndex to stop, using partialResult as input
if (done)
callback(err, result);
else
process.setImmediate ( go look this part up or I'll edit tomorrow)...
But the idea is you call youself again with start += 100.
}
In between every 100 calculations node will have time to process other requests, handle other callbacks, etc. Of course, if they trigger another huge calculation evedntually things will grind to a halt.

Big json(like 100MB) to excel in javascript [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
I'am new to REST technology. I'm currently dealing with a json response from a server and presenting the data on the client side.
I am now getting around 22MB of json data, and I need to export it to an excel sheet.
My problem:
When I'm iterating to json , the browser is not responding. What am I doing wrong?

IDEALLY, you should not be exporting such a big excel file for download. You must be breaking the data to chunks and asking user to download in batches. However, if it may be a part of your requirement, to handle it, refer answer below.
You might have to download the json in batches and combine as a single excel file and ask user to download!
The unresponsive script dialog box shows when some javascript thread takes too long too complete. Editing the registry could work, but you would have to do it on all client machines. You could use a "recursive closure" as follows to alleviate the problem. It's just a coding structure in which allows you to take a long running for loop and change it into something that does some work, and keeps track where it left off, yielding to the browser, then continuing where it left off until we are done.
Figure 1, Add this Utility Class RepeatingOperation to your javascript file. You will not need to change this code:
RepeatingOperation = function(op, yieldEveryIteration) {
//keeps count of how many times we have run heavytask()
//before we need to temporally check back with the browser.
var count = 0;
this.step = function() {
//Each time we run heavytask(), increment the count. When count
//is bigger than the yieldEveryIteration limit, pass control back
//to browser and instruct the browser to immediately call op() so
//we can pick up where we left off. Repeat until we are done.
if (++count >= yieldEveryIteration) {
count = 0;
//pass control back to the browser, and in 1 millisecond,
//have the browser call the op() function.
setTimeout(function() { op(); }, 1, [])
//The following return statement halts this thread, it gives
//the browser a sigh of relief, your long-running javascript
//loop has ended (even though technically we havn't yet).
//The browser decides there is no need to alarm the user of
//an unresponsive javascript process.
return;
}
op();
};
};
Figure 2, The following code represents your code that is causing the 'stop running this script' dialog because it takes so long to complete:
process10000HeavyTasks = function() {
var len = 10000;
for (var i = len - 1; i >= 0; i--) {
heavytask(); //heavytask() can be run about 20 times before
//an 'unresponsive script' dialog appears.
//If heavytask() is run more than 20 times in one
//javascript thread, the browser informs the user that
//an unresponsive script needs to be dealt with.
//This is where we need to terminate this long running
//thread, instruct the browser not to panic on an unresponsive
//script, and tell it to call us right back to pick up
//where we left off.
}
}
Figure 3. The following code is the fix for the problematic code in Figure 2. Notice the for loop is replaced with a recursive closure which passes control back to the browser every 10 iterations of heavytask()
process10000HeavyTasks = function() {
var global_i = 10000; //initialize your 'for loop stepper' (i) here.
var repeater = new this.RepeatingOperation(function() {
heavytask();
if (--global_i >= 0){ //Your for loop conditional goes here.
repeater.step(); //while we still have items to process,
//run the next iteration of the loop.
}
else {
alert("we are done"); //when this line runs, the for loop is complete.
}
}, 10); //10 means process 10 heavytask(), then
//yield back to the browser, and have the
//browser call us right back.
repeater.step(); //this command kicks off the recursive closure.
};
Adapted from this source:
http://www.picnet.com.au/blogs/Guido/post/2010/03/04/How-to-prevent-Stop-running-this-script-message-in-browsers

How to run javascript function in "background" / without freezing UI

I've done an HTML form which has a lot of questions (coming from a database) in many different tabs. User then gives answers in those questions. Each time a user changes a tab my Javascript creates a save. The problem is that I have to loop through all questions each time the tab is changed and it freezes the form for about 5 seconds every time.
I've been searching for an answer how I can run my save function in the background. Apparently there is no real way to run something in the background and many recommend using setTimeout(); For example this one How to get a group of js function running in background
But none of these examples does explain or take into consideration that even if I use something like setTimeout(saveFunction, 2000); it doesn't solve my problem. It only postpones it by 2 seconds in this case.
Is there a way to solve this problem?

You can use web workers. Some of the older answers here say that they're not widely supported (which I guess they weren't when those answers were written), but today they're supported by all major browsers.
To run a web worker, you need to create an instance of the built-in Worker class. The constructor takes one argument which is the URI of the javascript file containing the code you want to run in the background. For example:
let worker = new Worker("/path/to/script.js");
Web workers are subject to the same origin policy so if you pass a path like this the target script must be on the same domain as the page calling it.
If you don't want to create an new Javascript file just for this, you can also use a data URI:
let worker = new Worker(
`data:text/javascript,
//Enter Javascript code here
`
);
Because of the same origin policy, you can't send an AJAX request from a data URI, so if you need to send an AJAX request in the web worker, you must use a separate Javascript file.
The code that you specify (either in a separate file or in a data URI) will be run as soon as you call the Worker constructor.
Unfortunately, web workers don't have access to neither outside Javascript variables, functions or classes, nor the DOM, but you can get around this by using the postMessage method and the onmessage event. In the outside code, these are members of the worker object (worker in the example above), and inside the worker, these are members of the global context (so they can be called either by using this or just like that with nothing in front).
postMessage and onmessage work both ways, so when worker.postMessage is called in the outside code, onmessage is fired in the worker, and when postMessage is called in the worker, worker.onmessage is fired in the outside code.
postMessage takes one argument, which is the variable you want to pass (but you can pass several variables by passing an array). Unfortunately, functions and DOM elements can't be passed, and when you try to pass an object, only its attributes will be passed, not its methods.
onmessage takes one argument, which is a MessageEvent object. The MessageEvent object has a data attribute, which contains the data sent using the first argument of postMessage.
Here is an example using web workers. In this example, we have a function, functionThatTakesLongTime, which takes one argument and returns a value depending on that argument, and we want to use web workers in order to find functionThatTakesLongTime(foo) without freezing the UI, where foo is some variable in the outside code.
let worker = new Worker(
`data:text/javascript,
function functionThatTakesLongTime(someArgument){
//There are obviously faster ways to do this, I made this function slow on purpose just for the example.
for(let i = 0; i < 1000000000; i++){
someArgument++;
}
return someArgument;
}
onmessage = function(event){ //This will be called when worker.postMessage is called in the outside code.
let foo = event.data; //Get the argument that was passed from the outside code, in this case foo.
let result = functionThatTakesLongTime(foo); //Find the result. This will take long time but it doesn't matter since it's called in the worker.
postMessage(result); //Send the result to the outside code.
};
`
);
worker.onmessage = function(event){ //Get the result from the worker. This code will be called when postMessage is called in the worker.
alert("The result is " + event.data);
}
worker.postMessage(foo); //Send foo to the worker (here foo is just some variable that was defined somewhere previously).

Apparently there is no real way to run something on background...
There is on most modern browsers (but not IE9 and earlier): Web Workers.
But I think you're trying to solve the problem at the wrong level: 1. It should be possible to loop through all of your controls in a lot less than five seconds, and 2. It shouldn't be necessary to loop through all controls when only one of them has changed.
I suggest looking to those problems before trying to offload that processing to the background.
For instance, you could have an object that contains the current value of each item, and then have the UI for each item update that object when the value changes. Then you'd have all the values in that object, without having to loop through all the controls again.

You could take a look at HTML5 web workers, they're not all that widely supported though.

This works in background:
setInterval(function(){ d=new Date();console.log(d.getTime()); }, 500);

If you can't use web workers because you need to access the DOM, you can also use async functions. The idea is to create an async refreshUI function that refreshes the UI, and then call that function regularly in your function that takes long time.
The refreshUI function would look like this:
async function refreshUI(){
await new Promise(r => setTimeout(r, 0));
}
In general, if you put await new Promise(r => setTimeout(r, ms)); in an async function, it will run all the code before that line, then wait for ms milliseconds without freezing the UI, then continues running the code after that line. See this answer for more information.
The refreshUI function above does the same thing except that it waits zero milliseconds without freezing the UI before continuing, which in practice means that it refreshes the UI and then continues.
If you use this function to refresh the UI often enough, the user won't notice the UI freezing.
Refreshing the UI takes time though (not enough time for you to notice if you just do it once, but enough time for you to notice if you do it at every iteration of a long for loop). So if you want the function to run as fast as possible while still not freezing the UI, you need to make sure not to refresh the UI too often. So you need to find a balance between refreshing the UI often enough for the UI not to freeze, but not so often that it makes your code significantly slower. In my use case I found that refreshing the UI every 20 milliseconds is a good balance.
You can rewrite the refreshUI function from above using performance.now() so that it only refreshes the UI once every 20 milliseconds (you can adjust that number in your own code if you want) no matter how often you call it:
let startTime = performance.now();
async function refreshUI(){
if(performance.now() > startTime + 20){ //You can change the 20 to how often you want to refresh the UI in milliseconds
startTime = performance.now();
await new Promise(r => setTimeout(r, 0));
}
}
If you do this, you don't need to worry about calling refreshUI to often (but you still need to make sure to call it often enough).
Since refreshUI is an async function, you need to call it using await refreshUI() and the function calling it must also be an async function.
Here is an example that does the same thing as the example at the end of my other answer, but using this method instead:
let startTime = performance.now();
async function refreshUI(){
if(performance.now() > startTime + 20){ //You can change the 20 to how often you want to refresh the UI in milliseconds
startTime = performance.now();
await new Promise(r => setTimeout(r, 0));
}
}
async function functionThatTakesLongTime(someArgument){
//There are obviously faster ways to do this, I made this function slow on purpose just for the example.
for(let i = 0; i < 1000000000; i++){
someArgument++;
await refreshUI(); //Refresh the UI if needed
}
return someArgument;
}
alert("The result is " + await functionThatTakesLongTime(3));

This library helped me out a lot for a very similar problem that you describe: https://github.com/kmalakoff/background
It basically a sequential background queue based on the WorkerQueue library.

Just create a hidden button. pass the function to its onclick event.
Whenever you want to call that function (in background), call the button's click event.
<html>
<body>
<button id="bgfoo" style="display:none;"></button>
<script>
function bgfoo()
{
var params = JSON.parse(event.target.innerHTML);
}
var params = {"params":"in JSON format"};
$("#bgfoo").html(JSON.stringify(params));
$("#bgfoo").click(bgfoo);
$("#bgfoo").click(bgfoo);
$("#bgfoo").click(bgfoo);
</script>
</body>
</html>

Signal the end of a series of (n) AJAX calls

I have a web application which crawls sites in a CMS and looks for data of a certain type.
It works a lot like a recursive file/directory loop:
//pseudo code
var rootWeb = context.site.rootWeb();
var objectThatHoldsAllResults;
recursiveSiteSearch(rootWeb);
function recursiveSiteSearch(webSite) {
//Get all content of a certain type and add to objectThatHoldsAllResults
//Get all SubSites and throw them into a loop that runs recursiveSiteSearch
}
This application lives in the cloud, and has no idea how many subsites live in each CMS that accesses it.
Each time the loop gets all content of a certain type, it makes an AJAX call to the website.
I need to know when the recursion is done, but have no idea how to do so.

Straightforwardly, recursion will have ended when execution falls through to the statement following recursiveSiteSearch(rootWeb);.
However, asynchronicity (ajax) within recursiveSiteSearch may/will(?) mean that some latent activity still exists at that point.
Therefore, you appear to need a mechanism for detecting when all promises (ie. all ajax requests initiated within the recursion) are complete.
jQuery, provides such a mechanism.
pseudo code :
function recursiveSiteSearch(webSite) {
//Get all content of a certain type and add to objectThatHoldsAllResults
//Get all SubSites and throw them into a loop that runs recursiveSiteSearch
//Within the loop, push jqXHR objects onto the externally declared `promises` array.
}
var rootWeb = context.site.rootWeb();
var objectThatHoldsAllResults;
var promises = [];
recursiveSiteSearch(rootWeb);
jQuery.when.apply(jQuery, promises).done(function() {
//statements here will execute when
//recursion has finished and all ajax
//requests have completed.
});
The reason this should work is that jqXHR objects (returned by jQuery.ajax() and its shorthand forms) implement jQuery's Promise interface.