Error traversing large arrays in JavaScript without timeout - javascript

When I'm am executing the following code in Node.js, I get the folliowing error:
RangeError: Maximum call stack size exceeded
This is the code:
var arr = [];
for (var i = 0; i <= 1000000; i++)
arr.push(i);
function nextStep(index) {
if (index === arr.length) {
return;
} else {
console.log(index);
}
nextStep(++index);
}
nextStep(0);
I have no clue what is happening, but near index = 17938, the execution terminates.
Using a setTimeout() helps. What could be wrong here?

You are entering a recursive function. This means that the first function won't return until all the other functions have returned. So if you had four items,
fn(item1)
calls -> fn(item2)
calls -> fn(item3)
calls -> fn(item4)
As you can see, the nesting builds up and up. This is called the stack. There is a maximum size for the stack, to prevent infinite recursion and runaway processes. You have found it with 17938.
This is an inherent flaw with recursion. It can be a stylish way to approach a task, but it has its limitations. The best way to correct it is to use a loop instead:
for (var i = 0; i < arr.length; i++) {
Using setTimeout also works, because the function is not called from the function itself, but is executed with a new stack. However, it will have significantly lower performance than a loop or than a normal recursive function.

There's a certain number of calls that can be made, based on the call stack of different browsers. Most likely, you are testing your code in Chrome, since, I believe, it has the call stack near 20.000. Your code will execute the nextStep function more than 20.000 times (1 million), which means that if your function will not return something until it reaches the call stack limit of that particular browser, it will generate the error that you are getting.

Calling nextStep inside nextStep causes a stack overflow since you never return from the function (unless you found the end of the array, and if the array is too big you will never reach it until the stack overflows).
An example:
You are going to move all the stones from one place to another. Your function is like going to the place where the stones are and pick up one to be delivered to another place. But before you can deliver the stone, you must pick up another, and before you can deliver that you need to pick up another... and soon you are carrying 17938 stones. That is a little bit to heavy and you get crushed under all the stones. (or in the javascript case you get an exception)
When you use setTimetout it is like you are going to the place, pick up a stone, and make a note that you should pickup another stone. Then you deliver the stone. After that you look at your notes and see that you should pick up at stone. And you can do this 1000000 times since you are only carrying one stone at a time.

Related

Javascript, maximun call stack size exceeded

I was writing some code for a widget in "Scriptable" that shows random word at a certain time. The widget calls itself with a timeout, on iPad. But it's exceeding the stack size. How do I solve this? I am using Javascript within "Scriptable" framework, so I don't have much freedom.
kanjiObj: Object; kanjiKeys: List; Timer().schedule(timeinterval, repeat, callback)
var randomNum = Math.floor(Math.random()*150)+1
var kanji = kanjiKeys[randomNum]
var kanjiMeaning = kanjiObj[kanjiKeys[randomNum]]
if(config.runsInWidget){
let widget = createWidget()
Script.setWidget(widget)
Script.complete()
function createWidget(){
let widget = new ListWidget()
widget.addText(kanji + "=>" + kanjiMeaning)
widget.wait = new Timer().schedule(1000*60*60, 150, createWidget())
In your case, you are calling the createWidget() function recursively by mistake. Your intention was different, you wanted to pass the function as a parameter to the schedule function, which accepts a function callback as the last parameter.
So you have to change to
widget.wait = new Timer().schedule(1000*60*60, 150, () => createWidget())
or
widget.wait = new Timer().schedule(1000*60*60, 150, createWidget)
But since this question is about "call stack size exceeded" people who have a different kind of this problem could stumble upon it, so I will answer in a general way below as well.
Programming is about recognizing patterns, so what you have in your code is a pattern (structure) of an infinite recursive call
function someFunction() {
// ... code before the recursive call that never returns from this function
// an unconditional call (or a conditional chain that is always true)
// of the function itself in the function scope instead of inside a callback/lambda
someFunction()
// ... could be more code, which does not matter,
// until the pattern above still holds true
}
So a structure (pattern) like this will result in an infinite recursive call, which eventually ends with a maximum call stack size exceeded. Call stack size is limited and an infinite recursive call eventually fills it up.
Javascript does not have built-in tail-recursion optimization (in practice, see compatibility), but in programming languages that do, if the function is tail-recursive, it would be optimized to a loop. In the case of an infinite tail-recursive function it would be optimized to an infinite loop, which would not fill the call stack, it would just loop forever (it would be working, but if not implemented correctly, it could make the application unresponsive).
So to resolve a problem like this in general, one has to break this pattern by breaking any (at least one) of the conditions above. This is the list:
code before the recursive call that never returns from this function
add a conditional return statement (also called a stop condition)
an unconditional call (or a conditional chain that is always true) to the function itself in the function scope instead of inside a callback/lambda
make the call conditional (make sure the condition chain can be false) or put it inside a callback/lambda. When you put it inside a
callback/lambda, then a different pattern applies (you have to check
the logic inside the call that will be calling the callback), but
calling the callback still has to be conditional, it has to have a
limit at some point.
after making a change, the code that is after the call, needs to be checked for the same pattern again, if the pattern is there again, break it in a similar way. Keep repeating this until the whole function does not form the pattern anymore - has stop condition(s) where needed.
In cases when you do need an infinite recursive call and the function is tail-recursive and your language does not support tail-recursion optimization, you would need to do the tail optimization yourself or use a library/framework that lets you do that.
If this does not solve your problem, have a look at the answers in this question that has collected a lot of reasons why a "maximum call stack size exceeded" might happen in Javascript.

try/catch - How many recursive calls? call stack size loop

I have this function which I want to loop without stopping the script.
But when I use 'console.log' the function stops. If I remove it, the function continues.
How to make this function continue after it has exceeded stack size?
var i = 0;
function computeMaxCallStackSize() {
try {
i++;
console.log(i);
//some function to execute
computeMaxCallStackSize();
} catch (e) {
//'e' is the error when max stack exceeds
// setTimeout() is to wait before calling the function
// if stack size exceeds it recalls the function
setTimeout(computeMaxCallStackSize(), 0);
}
}
computeMaxCallStackSize();
Edit:
This function will cause a stack overflow and continues.
var a = 1;
var b = 8769; //stack size
func(a);
function func(a) {
try {
console.log(a);
if(a % b === 0){
setTimeout( function() {func(a+1); }, 0);
} else {
func(a+1);
}
} catch (e) {
console.error(e);
setTimeout( function() {func(a); }, 1);
}
}
If I remove it, the function continues.
No it doesn't, it will finish also, but much later. console.log() taking much more resources cause it needs to render text in console, so it's reaching call stack faster.
You just made infinite loop, that will run till browser will run out of memory.
I don't see a way to catch stackoverflow in nodejs. From what I know from other ecosystems, it is impossible. See C# catch a stack overflow exception for example. Also see a note about exceptions:
Some exceptions are unrecoverable at the JavaScript layer. Such
exceptions will always cause the Node.js process to crash. Examples
include assert() checks or abort() calls in the C++ layer.
Process stack is fixed-size, contiguous block of memory, usually not very big (because it's purpose is to make use of process caches). This means that you can't "extend" it on your wish (only at process start). If you can't extend it, why bothering checking it's boundaries every time you do a function call? OS will do the job and crash your process for you!
I know it's not always possible in case of highly-recursive academic algorithms, but from business perpective, you should always unwrap your recursion, especially if it's deep. Even if you think you control it (like, it's only 2000 deep, who cares?), in reality you don't, since usually you don't fully control the caller stack size, and the end-user platform limitations.
BUT!
If you REALLY want to recover your process after stackoverflow, and you're ready for such an adventure (win api, build node from sources, etc.), I will leave this for you.
Your setTimeout is kind of pointless the way it is:
// Current code:
setTimeout(computeMaxCallStackSize(), 0);
// What you really want:
setTimeout(computeMaxCallStackSize, 0);
To answer your question - you don't want to write code which can exceed the stack - that's always an indication of poorly written code. Furthermore, there's not really a way to check the stack size or to check when the stack is "clear" again (for good reason - people would abuse that knowledge to write poor code).
If you need recursion, you almost always build in a "depth" parameter to the function so the function stops after a certain call depth. Something like this:
function someRecursiveFunction(_depth = 0) {
if(_depth < 100) {
// do the stuff
someRecursiveFunction(_depth + 1)
}
throw new Error('Oh no, we called this function too many times');
}
If you're just doing this for giggles and really want to see how deep the stack hole goes, I guess you could count the iterations. As mentioned in other comments, the console log is going to take up additional resources - so you might get 1 million iterations without the logging and only 100K with logging. Again, there's no way to ask the JS runtime "how many resources to I have left". The function you are writing is simply a "for fun" experiment and should not be used in real life code.

Why is "Self Time" so high in an effectively empty function?

I have a computationally heavy function which is called many times in a loop:
function func() { // Some fluff
for(let i = 0; i < 1000; i++) {
i *= 10
i /= 10
}
}
function run() {
for(let i = 0; i < 100000; i++) {
func()
}
}
run()
When I profile this script using Chrome's DevTools, I get this:
run has a self time of 887ms out of the total time of 1015ms even though the only thing it does is repeatedly call func.
I would expect func to have the majority of self time since it's a leaf function.
Why is this?
(V8 developer here.)
the function was automatically inlined after some time when it became "hot".
Correct. As soon as run is optimized, the optimizer decides to inline func into it. After that, as far as the profiler is concerned, all the time is spent in run.
(To verify this, run the snippet in d8 or node with --trace-turbo-inlining.)
Side note: getting to optimized code for run takes a bit longer than usual in this case, because the function never returns to get called again (which is the best time to switch to optimized code). The system waits a bit for that to happen, and when it doesn't happen, run is eventually "on-stack replaced". This is a typical pattern that occurs frequently in small tests and benchmarks, and rarely in real-world code.
Doesn't this just show that performing 100000 function calls are more expensive than a 1000 iterations of two simple arithmetic operations -- which makes sense?
No, it doesn't show that; that's just one particular way how one could be misled by this microbenchmark.
Personally, I'm mildly disappointed to see (with --print-opt-code) that the compiler didn't realize that i *= 10; i /= 10; is a no-op and could be dropped entirely. That would have been another great way to be misled here. Oh well, there's probably some reason why it's harder than it looks for the compiler to figure out that that transformation would be both applicable and safe...

Avoiding stack overflow by using setTimeout

I've found the following question here:
The following recursive code will cause a stack overflow if the array
list is too large. How can you fix this and still retain the recursive
pattern?
And the answer:
The potential stack overflow can be avoided by modifying the
nextListItem function as follows:
var list = readHugeList();
var nextListItem = function() {
var item = list.pop();
if (item) {
// process the list item...
setTimeout( nextListItem, 0);
}
};
The stack overflow is eliminated because the event loop handles the
recursion, not the call stack. When nextListItem runs, if item is not
null, the timeout function (nextListItem) is pushed to the event queue
and the function exits, thereby leaving the call stack clear. When the
event queue runs its timed-out event, the next item is processed and a
timer is set to again invoke nextListItem. Accordingly, the method is
processed from start to finish without a direct recursive call, so the
call stack remains clear, regardless of the number of iterations.
Can somebody please explain to me:
whether this use case is ever practical
why long array can cause stack overflow
This is just a hacky alternative to trampolines, which in turn are just a hacky alternative to TCO.
When you call a function in Javascript, you add a frame to the call stack. That frame contains information about the variables in the scope of the function and how it was called.
Before we call a function, the call stack is empty.
-------
If we call function foo, then we add a new frame to the top of the stack.
| foo |
-------
When foo finishes executing, we pop the frame off the stack again, leaving it empty again.
Now, if foo in turn calls another function bar, then we'll need to add a new frame onto the stack, whilst foo is executing.
| bar |
| foo |
-------
Hopefully you can see that if a function calls itself recursively it keeps adding new frames to the top of the call stack.
| ... |
| nextListItem |
| nextListItem |
| nextListItem |
| nextListItem |
----------------
Recursive functions will keep adding frames until either they finish processing, or they exceed the max length of the call stack, resulting in an overflow.
Because setTimeout is an asynchronous operation, it doesn't block your function, which means nextListItem will be allowed to finish and its frame can be popped off the call stack—preventing it from growing. The recursive call will be handled with the event loop instead.
Is this pattern ever useful? The max size for the call stack depends on your browser, but it can be as low as 1130. If you wanted to process an array with a few thousand elements using a recursive function, then you'd risk blowing the call stack.
Trampolines use a similar technique, but rather than offloading the work to the event loop, you return a function which calls the next iteration instead, then the calls can be managed with a while loop (which doesn't affect the stack).
var nextListItem = function() {
var item = list.pop();
if (item) {
// process the list item...
return nextListItem;
}
};
while(recur = recur()) {}
It normally isn't, but in the off chance you decide you need to recursively chain the same function call for long sequences this could come in handy.
Stack overflow during recursive operations occurs when the amount of stack memory allocated for a particular program has been fully utilized. A sufficiently long array that is traversed recursively can cause a stack overflow. Perhaps you do not understand how the call stack works?
The repeating for loop is the most efficient for the code posted. However, if your actual code cannot be fitted to using a for loop, then there is another alternative.
The use of setTimeout depends on your opinion of 'practical,' so let's just list the facts so you can decide for yourself.
setTimeout forces the browser to swamp the CPU with operations to get millisecond precision timing. This can be very power inefficient.
With setTimeout, every iteration will cost 4ms. Just 4 iterations will take the time for a whole frame to get rendered. 250 iterations, and a whole second goes by.
But there is an alternative to setTimeout that will help you achieve exactly what you are trying to do without using setTimeout: the DeferStackJS library. If you use DeferStackJS, then all you would need to do is as follows.
var list = readHugeList();
var nextListItem = function() {
var item = list.pop();
if (item) {
// process the list item...
DeferStack( nextListItem );
}
};
Please emphasize that the above snippet of code is intended for demonstrating how to integrate DeferStackJS. Truth be told, using either DeferStackJS or Array.prototype.pop would be very inappropriate for this specific snippet of code. Instead, the following code would beat both of them hands down.
var list = readHugeList();
var nextListItem = function() {
"use strict";
var item, i = list.length;
while (i--) { // Bounds check: as soon as i === -1, the while loop will stop.
// This is because i-- decrements i, returning the previous
// value of i
item = list[i];
if (!item) break; // break as soon as it finds a falsey item
// Place code here to be executed if it found a truthy value:
// process the list item...
}
if (i !== -1) {
// Place code here to be executed if it found a falsey value:
}
};
The only reason DeferStackJS is mentioned is because I am a firm believer in that the #1 duty of a person answering a forum is to answer the original question asked. Then after they answer that they can put up commentary second-guessing the question about what was intended to be asked like this.

A "too much recursion" error in Firefox only sometimes?

I have a pretty simple thing I'm doing with javascript and basically only sometimes will javascript give me a "too much recursion" error.
The code in question:
if(pageLoad===undefined){
var pageLoad=function(){};
}
var pageLoad_uniqueid_11=pageLoad;
var pageLoad=function(){
pageLoad_uniqueid_11();
pageLoad_uniqueid_12();
};
var pageLoad_uniqueid_12=function(){
alert('pageLoad');
};
$(document).ready(function(){
pageLoad();
});
(yes I know there are better way of doing this. This is difficult to change though, especially because of ASP.Net partial postbacks which aren't shown).
Anyway, when the too much recursion error happens, it continues to happen until I restart Firefox. when I restart Firefox it all works as normal again. How do I fix this?
I've also made a jsbin example
Update
Ok I've found out how to reliably reproduce it in our code, but it doesn't work for the jsbin example. If I create a new tab and go to the same page(have two tabs of the same address) and then refresh the first tab two times then I get this error consistently. We are not using any kind of session or anything else that I can think of that could cause such a problem to only occur in one tab!
Update 2
Not as reliable as I thought, but it definitely only occurs when more than one tab of the same page is open. It'll occur every few reloads of one of the tabs open
I've also updated my code to show an alert when pageLoad(the if statement) is initially undefined and when it is initially defined. Somehow, both alerts are showing up. This code is not duplicated in the rendered page and there is no way that it is being called twice. It is in a top level script element not surrounded by a function or anything! My code ends up looking like
if(pageLoad===undefined){
var pageLoad=function(){};
alert('new');
} else {
alert('old');
}
The code in question -- by itself -- should never result in an infinite recursion issue -- there is no function-statement and all the function objects are eagerly assigned to the variables. (If pageload is first undefined it will be assigned a No-Operation function, see next section.)
I suspect there is additional code/events that is triggering the behavior. One thing that may cause it is if the script/code is triggered twice during a page lifetime. The 2nd time pageload will not be undefined and will keep the original value, which if it is the function that calls the other two functions, will lead to infinite recursion.
I would recommend cleaning up the approach -- and having any issues caused by the complications just disappear ;-) What is the desired intent?
Happy coding.
This is just some additional info for other people trying to look for similar "too much recursion" errors in their code. Looks like firefox (as an example) gets too much recursion at about 6500 stack frames deep in this example: function moose(n){if(n%100 === 0)console.log(n);moose(n+1)};moose(0) . Similar examples can see depths of between 5000 and 7000. Not sure what the determining factors are, but it seems the number of parameters in the function drastically decrease the stack frame depth at which you get a "too much recursion" error. For example, this only gets to 3100:
function moose(n,m,a,s,d,fg,g,q,w,r,t,y,u,i,d){if(n%100 === 0)console.log(n);moose(n+1)};moose(0)
If you want to get around this, you can use setTimeout to schedule iterations to continue from the scheduler (which resets the stack). This obviously only works if you don't need to return something from the call:
function recurse(n) {
if(n%100 === 0)
setTimeout(function() {
recurse(n+1)
},0)
else
recurse(n+1)
}
Proper tail calls in ECMAScript 6 will solve the problem for some cases where you do need to return something from calls like this. Until then, for cases with deep recursion, the only answers are using either iteration, or the setTimeout method I mentioned.
I came across this error. The scenario in my case was different. The culprit code was something like this (which is simple concatenation recessively)
while(row)
{
string_a .= row['name'];
}
I found that JavaScript throws error on 180th recursion. Up till 179 loop, the code runs fine.
The behaviors in Safaris is exactly the same, except that the error it shows is "RangeError: Maximum call stack size exceeded." It throws this error on 180 recursion as well.
Although this is not related to function call but it might help somebody who are stuck with it.
Afaik, this error can also appear if you state a wrong parameter for your ajax request, like
$.getJSON('get.php',{cmd:"1", elem:$('#elem')},function(data) { // ... }
Which then should be
elem:$('#elem').val()
instead.
This will also cause the "too much recursion" issue:
class account {
constructor() {
this.balance = 0; // <-- property: balance
}
set balance( amount ) { // <-- set function is the same name as the property.
this.balance = amount; // <-- AND property: balance (unintended recursion here)
}
}
var acc = new account();
Using unique names is important.
Ok, so why is this happening?
In the set function it isn't actually setting the property to amount, instead it's calling the set function again because in the scope of the set function it is the same syntax for both setting the property AND calling the set function.
Because in that scope this is the same as account and (account OR this).balance = amount can both call the set function OR set the property.
The solution to this is to simply change the name of either the property or the set function in any way (and of course update the rest of the code accordingly).

Categories