We Keep Coding

How do you store activation records with complex lexical scopes and returning functions as results and such?

I have read in several places such as here about "Access Links" and "The Display Method" for looking up non-local variables. However, none seem to touch on the complicated situation like in JavaScript where you can have functions inside of functions, and return those functions from functions, all the while keeping alive the "internal scope" of those functions.
Here is a complicated nested function.
let m = foo()
m()
m()
function foo() {
let x = 10
function bar() {
let y = 2 ** x
function hello() {
let z = 30
function another() {
// PLACE_2
let q = x + y + z
function nested() {
q += (17 * x) - (3 * z)
// PLACE_1
console.log(q)
console.log(w)
}
return nested
}
// PLACE_3
let w = another()
return w
}
return hello
}
// PLACE_4
let a = bar()
let nested = a() // hello returns another
nested()
nested()
return nested
}
At PLACE_1, we are using variables x/z/q, but not y/a/nested. Basically we create all these lingering objects in the scope tree.
How does an activation record with "Access Links" or "Display Method" work in a situation like this? I am used to having functions call other functions, but not keep the environment of past functions around. How do these work in a complex system like this with scopes lingering?
I am trying to implement a compiler and am currently stuck on how to actually implement the Activation Records. I can do it where the activation records are a simple parent/child relationship based on calling order, but I don't see how to make it work with a situation like this. I want to have a sort of variable lookup mechanism, but I don't know what needs to be stored in a situation like this.
Reading something like this doesn't give any insight into real-world situations like this, how it's modeled under the hood.

Displays can only be used if the semantics of the language don't allow non-local variables to outlive their original scope, since the display is basically an efficient eay to find an activation record on the stack. If you want to implement true closures, you'll need to use a different mechanism which involves heap-allocation of enclosed variables. (Unless you can prove that the enclosed variable never escapes. That's often the case, but it can be hard to prove.)
AUUI, Javascript retains the entire activation record of the function call which created the closed variable. But that's dependent on the semantics of the language, and unnecessarily keeping active links to other variables in the activation record prevents them from being garbage collected in a timely manner. It should only be necessary to keep links to the variables actually being enclosed.
Note that there are two distinct cases (again, this depends on language semantics). In the simple case, the captured variable is immutable (although it might contain a mutable value). In that case, you can capture the value by adding it as a data member in the closure. (The closure object itself needs to be heap-allocated but you don't need to do that until the closure is actually created, which might never happen.)
In the more complicated case, the variable itself is mutable and potentially mutated by the inner function [Note 1]. In that case, the variable itself must be "boxed"; that is, enclosed in a container which is used as a handle for the variable. But the mutation might happen before the variable's scope terminates, in which case the change needs to be visible in that scope. If the variable is boxed, the outer function needs to be aware of that because access to the variable is indirect. So boxing the variable onto the heap is a cost which must be paid before it is known to be necessary (depending on the quality of the escape analysis).
That's where the Lua implementation is interesting, at least. It manages to avoid heap-allocating the box in many cases, but there is an additional cost when it does so. That seems to work out well for Lua use cases but I don't know whether it would be a good solution in your case.
Notes:
There's a very common case where a variable is mutable because its value cannot conveniently be computed in a single initialisation expression, but all mutations occur before the variable is captured. Such a variable can be treated as though it were immutable (or as though it was computed and then made the value of a different variable with the same name). That case is fairly easy to detect. (There are also variables whose last mutation is after capture but before the first invocation of a capturing function. Those can also usually be detected.) These variables can be captured as though they were constant, just putting the value into the closure.
Informal surveys indicate that the vast majority of captured variables are either constant or constant-after-capture, which could considerably reduce the need for heap-allocated boxes.

Specific real life application of closures in javascript

I have used JS for two years and my pluralsight accessment rates me as proficient in JS, I understand prototypical inheritance, higher order functions, IIFEs etc and I have used them in real world instances but closures is one of those concept that you understand but can't find any reason why you would want to use them in real world development, I mean I know that if I say,
function expClosure(val){
//do something to val-->anotherVal
return function(val){return anotherVal)};
}
var exp = expClosure(val);
exp(); --> returns the value of anotherVal;
My question is why would I want to do this, or rather what specific instances can lead me to consider using this.

The main benefit to closures is you can "partially apply" a function using a closure, then pass the partially applied function around, instead of needing to pass the non-applied function, and any data you'll need to call it (very useful, in many scenarios).
Say you have a function f that will need in the future 2 pieces of data to operate. One approach is you could pass both pieces in as arguments when you call it. The problem with this approach is if the first piece of data is available immediately, and the second isn't, you'll have to pass it around with f so it's in scope when you want to call the function.
The other option is to give the available data to the function immediately. You can create a closure over the data, have the function reference the external data, then pass the function around by itself.
Option 2 is much simpler.
You can also use closures to implement static variables in functions in languages that don't natively support them. Clojure (the language) implements it's memoize function by having it return a modified version of the passed function that holds a reference to a map representing argument/return value pairs. Before the function is run, it first checks if the arguments already exist in the map. If they do, it returns the cached return value instead of recalculating it.
(Adapted from my answer to another question)

I wrote a bit about this in my practical bachelor thesis(2.2.2)
Hiding variables is a valuable usage of closure. Compared to some other languages variables can't be declared as private, public, etc. in JavaScript, but using closure you can hide variables that can only be used internally. E.g.
function Car() {
var speed = 0;
return {
accelerate: function() {
speed++;
}
}
}
var car = new Car();
car.accelerate();
speed is accessible by accelerate due to closure, but is otherwise completely inaccessible.

Since this question doesn't demand a programmatic answer, I am adding an answer and not a comment.
The example you quoted in the question is, I agree of a closure, and I am sure having access to pluralsight lectures, you well understand closures. So the aforestated example is not the only use case of closures. The closures are the functions which remember the scope in which they were created.
An obvious example of this is a callback registering mechanism which every one uses in jQuery. There are closures everywhere, and many of us have been unknowingly writing closures.
So if you have used redux, you would know the entire concept is based on closure, i.e. encapsulating the data(called state of application). Closure provisions a concept of private variables used in OOPS supporting languages.
I am adding another example of closure below, so you might be able to relate.
function processOrder(amount){
var temp_order_id = +new Date() + 'USERID';
var afterPaymentCompleteOrder = function(status){//callback
//afterPaymentCompleteOrder is a closure as it remembers the scope in which it is created when it is being called. So when ivoked it remembers the value of temp_order_id
if(status){
complete_order_method(temp_order_id);
}
else
delete_order_details(temp_order_id);
}
start_payment_process(amount, afterPaymentCompleteOrder);
}

What happens to a JavaScript function when it is closured in as a variable?

I've read many questions on closures and JavaScript on SO, but I can't find information about what happens to a function when it gets closured in. Usually the examples are string literals or simple objects. See the example below. When you closure in a function, the original function is preserved even if you change it later on.
What technically happens to the function preserved in a closure? How is it stored in memory? How is it preserved?
See the following code as an example:
var makeFunc = function () {
var fn = document.getElementById; //Preserving this, I will change it later
function getOriginal() {
return fn; //Closure it in
}
return getOriginal;
};
var myFunc = makeFunc();
var oldGetElementById = myFunc(); //Get my preserved function
document.getElementById = function () { //Change the original function
return "foo"
};
console.log(oldGetElementById.call(document, "myDiv")); //Calls the original!
console.log(document.getElementById("myDiv")); //Returns "foo"

Thanks for the comments and discussion. After your recommendations, I found the following.
What technically happens to the function preserved in a closure?
Functions as objects are treated no differently than any other simple object as closured variables (such as a string or object).
How is it stored in memory? How is it preserved?
To answer this, I had to dig through some texts on programming languages. John C. Mitchell's Concepts in Programming Languages explains that closured variables usually end up in the program heap.
Therefore, the variables defined in nesting subprograms may need lifetimes that are of the entire program, rather than just the time during which the subprogram in which they were defined is active. A variable whose lifetime is that of the whole program is said to have unlimited extent. This usually means they must be heap-dynamic, rather than stack-dynamic.
And more specific to JavaScript runtimes, Dmitry Soshnikov describes
As to implementations, for storing local variables after the context is destroyed, the stack-based implementation is not fit any more (because it contradicts the definition of stack-based structure). Therefore in this case closured data of the parent context are saved in the dynamic memory allocation (in the “heap”, i.e. heap-based implementations), with using a garbage collector (GC) and references counting. Such systems are less effective by speed than stack-based systems. However, implementations may always optimize it: at parsing stage to find out, whether free variables are used in function, and depending on this decide — to place the data in the stack or in the “heap”.
Further, Dmitry shows a varied implementation of the parent scope in function closures:
As we mentioned, for optimization purpose, when a function does not use free variables, implementations may not to save a parent scope chain. However, in ECMA-262-3 specification nothing is said about it; therefore, formally (and by the technical algorithm) — all functions save scope chain in the [[Scope]] property at creation moment.
Some implementations allow access to the closured scope directly. For example in Rhino, the [[Scope]] property of a function corresponds to a non-standard property __parent__.

Does a (JS) Closure Require a Function Inside a Function

I'm having a little difficulty with the inherent concept of a closure. I get the basic idea, but here's the thing: I thought that, technically, there "is a closure" inside every Javascript function. To quote wikipedia:
In computer science, a closure (also lexical closure, function closure
or function value) is a function together with a referencing
environment for the nonlocal names (free variables) of that function.
Such a function is said to be "closed over" its free variables.
So since you can define variables inside a function, they are "closed off" to the rest of your code, and so I see that as a closure. Thus, as I understand it:
(function(){var a = 1;}())
Is a (not very useful) example of a closure. Or heck, even just this:
function(){var a = 1;}
But, I think my understanding might be wrong. Others are telling me that for something to be a closure it has to persist a state, and so since nothing persists beyond that code it's not really a closure. That suggests that you need to have:
function(foo){foo.a = 1;}(bar); // bar.a = 1
or even (to ensure un-modifiability):
function(foo){var a = 1; bar.baz = function() { return a}}(bar); // bar.baz() = 1
So, technically speaking (I know several of the examples are practically speaking pointless, but) which of the above examples are actually examples of closures. And does a closure just have to be a space (ie. inside a JS function) where variables can be stored that can't be accessed form outside, or is persistence a key part of a closure's definition?
EDIT
Just noticed the wiki definition for the "closures" tag on Stack Overflow:
A closure is a first-class function that refers to (closes over)
variables from the scope in which it was defined. If the closure still
exists after its defining scope ends, the variables it closes over
will continue to exist as well.
While the SO wiki is certainly no final authority, the first sentence does seem to correlate with my understanding of the term. The second sentence then suggests how a closure can be used, but it doesn't seem like a requirement.
EDIT #2
In case it isn't clear from the varying answers here, the wikipedia answer, and the tag answer, there does not seem to be a clear consensus on what the word "closure" even means. So while I appreciate all the answers so far, and they all make sense if you go with the author's definition of closure, what I guess I'm really looking for is ... is there any actual "authoritative" definition of the word (and then if so, how does it apply to all of the above)?

You're being led astray by a wrong assumption of where the word "closure" comes from.
In a language-theoretic context, the point of a closure is that the function can refer to variables declared outside its own definition. It is immaterial whether it has internal variables, or that the internal variables are not visible from outside. In other words it is about seeing out from the function to its definition environment, not about seeing in from outside the function.
Why the weird word, then? Look at the function in your last example:
bar.baz = function() { return a }
This function contains a mention of the variable a which is not defined in the function body itself. It is a "free" variable of the function body, sort of a "hole" in the definition. We cannot execute the function without knowing, by some extraneous means, what variable the identifier a in the body refers to. Forming a closure at run-time pairs this "open" function body with a reference to the appropriate variable, thereby closing the hole in the definition. And that's where the name comes from.
(If you want the completely technical explanation, the underlying concept is that of a "closed" term in the lambda-calculus, which means one that has no free variables. Only closed term have independent meanings. A closure is then the combination of a (usually compiled) non-closed piece of source code, together with the contextual information that lets it behave like it was a closed term, and therefore be executable).
Addendum: In the common idiom
function() {
var blah;
// some code here
}();
the point is not to get a closure (you will get one, of course, but it doesn't do anything interesting for you), but to create a local scope for the blah variable. A local scope is conceptually quite a different thing from a closure -- in fact most C-lookalikes other than Javascript will create them at every {} block, whereas they may or may not have closures at all.

None of your samples are closures technically speaking. (But forth sample can be classified as such in some circumstances, see below)
Closure is a data structure that combines reference to a function and non-empty list of call frames (or scopes) active at the moment of declaration.
Closure is created by executing some code that contains declaration of a function that uses variables from outer scopes. In this case runtime, while executing the code, has to create not just a reference to the function but closure structure - function reference and reference to its current environment - list of call frames that hold used outer variables.
For example in my TIScript call frames are replaced on stack - when you exit from a function its call frame that includes collection of variables it uses is purged from the stack. Closure creation in my case happens when: VM meets function declaration instruction and that function is marked (by compiler) as the one that uses outer variables. In this case current chain of call frames that hold used variables is moved from stack to the heap - converted to GCable data objects and reference to the function and its call chain is stored as a reference.
Your fourth case physically does not require closure to be created - no need to store call frames for later use - bar.baz contains just a number - not a reference to function.
But this:
function(foo){
var a = 1;
bar.baz = function() { return a; };
}
creates closure in bar.baz field. When you later invoke bar.baz() function code is executed and value of 'a' variable will be taken from reference to outer call frame that is stored in closure.
Hope it clears something for you.

Closures in JavaScript (and other languages) are used to control and define scope. There's no requirement that you define a function within a function for it to "qualify" as a closure. The body of a function is a Closure. One of the more common uses is to declare a local scope variable that becomes a Private or Hidden member of some other object or function you'll return, but that's not a hard-fast rule.

Javascript performance with closure

var name = function(n) {
var digits = ['one','two','three','four'];
return digits[n];
}
var namenew = (function() {
digits = ['one','two','three','four'];
return function(n) {
return digits[n];
}
}());
Both the versions result in the same output, however it is said that the second version is much faster than the first version.
As I understand, first version executes the function everytime where as the second version stores the result of execution. That is what confuses me as a functional/regular OOPS programmer.
How can one save a function with its inner context? What is happening under the hood? Can some one pls clarify?

The real answer to that question would be about 3 pages long. But I try to make it as short as possible. ECMA-/Javascript is all about Execution Contexts and Object. There are three basic types of Context in ECMAscript: Global context, Function contexts and eval contexts.
Every time you call a function, your engine will spawn it in it's own function context. Also, there is a such called Activation object created. This mystic object is part of a function context which consists out of at least:
[[Scope chain]]
Activation object
"this" context value
There may be more properties on different engines, but these three are required for any implementation of ES. However, back to the topic. If a function context is invoked, all parent contexts (or more precisly, the Activation objects from the parent contexts) are copied over into the [[Scope]] property. You can think of this property like an array, which holds (Activation-) Objects. Now, any function related information is stored in the Activation object (formal parameters, variables, function declarations).
In your example, the digits variable is stored in the Activation object for namenew. The second when the inner anonymous function is created, it adds that Activation object into its [[Scope]] propertys. When you call digits[n] there, Javascript first tries to find that variable in its own Activation object. If that fails, the search goes into the Scopechain. And voila, there we found the variable because we copied the AO from the outer function.
I already wrote too much for a short answer, but to really give a good answer to such a question you have to explain some basic knowledge about ES here. I guess that is just enough to give you an idea what really happens "under the hood" (there is a lot more to know, if you want to read more I'll give you some references).
You asked for it, you get it:
http://dmitrysoshnikov.com/ecmascript/javascript-the-core/

The first function recreates digits every time it's executed. If it's a large array this is needlessly expensive.
The second function stores digits in a context shared only with namenew. Every time namenew is executed it only performs a single operation: return digits[n].
An example like this wont show any noticeable gains in performance, but with very large arrays/objects/function calls performance will be improved significantly.
In an OOP perspective using a closure in this manner is similar to storing data in a static variable.
Don't forget that namenew is receiving the result of the closure function. The closure itself is only executed once.