We Keep Coding

Trace the execution of ALL Javascript in a web app

Here is the situation: A complex web app is not working, and it is possible to produce undesired behavior consistently. The cause of the problem is not known.
Proposal: Trace the execution paths of all javascript code. Essentially, produce two monstrous logs which can then be fed into a diff algorithm to determine where the behavior related to the bug begins to diverge (as the cause is not apparent from application behavior, and both comprehending and obtaining a copy of the actual JS code being run is difficult, due to the many pages that must be switched to and copied out from the web inspector. Making it difficult is the fact that all pages are dynamically spliced together with Perl code, where significant portions of JS code exist only as (dynamic...) Perl strings).
The Web Inspector in Chrome does not have an option that I know about for logging an execution trace. Basically what I would like is a log of every line of JS that is executed, in the order that they are executed. I don't see this as being a difficult thing to obtain given that the JS VM is single-threaded. The problem is simply that the existing user-facing tools are not designed for quite this much hardcore debugging. If we look at the Profiler in the Dev Tools, it's clearly capable of the kind of instrumentation that I need, but it is fundamentally designed to do profiling instead of tracing.
How can I get started with this? Is there some way I can build Chrome from source, where I can
switch off JIT in V8?
log every single javascript expression evaluated by V8 to a file
I have zero experience with the development side of Chrome. So e.g. links to dev-builds/branches/versions/distros of Chrome/Chromium/Canary (what's the difference?) are welcome.
At this point it appears that instrumenting the browser with powerful js tracing is still likely to be easier than redesigning the buggy app. The architecture of the page is a disaster, but the functionality is complex, and it almost fully works. I just have to find the one missing piece.
Alternatively, if tools of this sort already exist, what are some other keywords I can search for them with? "Code Tracing" is pretty much the only thing I can come up with.
I tested dynaTrace, which was a happy coincidence as our app supports IE (indeed Chrome support just came out of beta), but this does not produce a text dump, it basically produces a massive Win32 UI expando-tree, which is impossible to diff. This makes me really sad because I know how much more difficult it was to make the representation of the trace show up that way, and yet it turns out being almost utterly useless. Who's going to scroll up and down that tree view and see anything really useful in it, in anything other than a toy example of a web app?

If you are developing a big web app, it is always good to follow a test driven strategy for the coding part of it. Using just a few tips allows you to make a simple unit testing script (using QUnit) to test pretty much all aspects of your app. Here are some potential errors and some ways of solving them.
Make yourself handlers to register long living Objects and a handler to close them the safe way. If the safe way does not succeed then it is the management of the Object itself failing. One example would be Backbone zombie views. Either the view has bad code in the close section, the parent close is not hooked or an infinite loop happened. Testing all view events is also good, although tedious.
By putting all the code for data fetching inside a certain module (I often use a bunch of Backbone.Model objects for each table/document in my DB) and handlers for each using a reqres pattern, you can test them 1 by 1 to see if they all fetch and save correctly.
If complex computation is needed, abstract it in a function or module so that it can be easily tested with known data.
If your app uses data binding, a good policy is to have a JSON schema for all data to be tested against your views containing your bindings. Check against the schema all the data required. This is applied to your Backbone.Model also.
Using a good IDE also helps. PyCharm (if you use Python for backend) or WebStorm are extremely good for testing and developing JavaScript/CoffeeScript. You can breakpoint and study your code at specific locations, inside your browser! Also it runs your code for auto-completion and you can see some of the errors that way.
I just cannot encourage enough the use of modules in your code. Although there is no JavaScript official way of doing it (next ECMAScript draft has it), you can still use good libraries for it. Good ones are: RequireJS, CommonJS or Marionette.Module (if you use Marionette as your framework). I think Ember/AngularJS also offers this kind of functionality but I did not work with them personally so I am not sure.
This might not give you an immediate solution to your problem and I don't think (IMO) there is an easy one either. My focus was to show you ways to develop so that errors can be easily spotted and countered, and all of it (depending on your Unit Testing) during development phase. Errors will always happen, as much as our programmer ego wants us to believe the contrary. Hope I helped :)

I would suggest a divide and conquer strategy, first via logging, and second via code. Wrap suspect methods of the code with console logging in and out events and when the bug occurs hopefully it is occurring between or after some event. If event logging will not shed light, bring parts of the code into a new page/app. Divide and conquer will find when the error starts occurring.

So it seems you're in the realm of weird already, so I'm thinking of a weird solution. I know nothing about the backend of chrome myself so we're in the same boat, but if you're feeling bold here's an idea. Maybe you could find/replace every newline in your javascript files with a piece of code that logs either to a global string or to the console a) what file you're in, b) the contents of "this" or something useful to you, and maybe even c) the time. This would at least get you started. Make sure it's wrapped in something distinct so you can just as easily remove it.

AMD Module Shaping: How to load only one JS function?

Many JavaScript libraries have a Builder tool that will allow you to "shape" what features of the library you depend on, both in terms of download bandwidth cost to the client and in terms of isolating functionality you actually need.
For example, I like many things in sugar.js, but I simply don't need or want katakana and hiragana character set handling. As the most trivial example, I want to be able to "shape" sugar.js to only export string.isBlank().
Is there a tool available to do this for me? Is there any ongoing effort by the EcmaScript committee to do something like this in a future version of JavaScript? Do any higher-level languages like TypeScript and CoffeeScript, offer hidden support for such "shaping"? I can do such "shaping" in C# for .NET DLLs via monolinker.
Basically, it looks to me like AMD handles the Loader aspect of a modern compiler, but does not handle the Linker aspect. Builders for jquery and dojo only work for a specific module, and aren't true Linkers, just Builders.
UPDATE: Google Closure Compiler is a compiler that takes JavaScript as input and produces JavaScript as output. The Advanced Compilation and Externs documentation suggests there is an API call to do this:
If putting together these export statements seems too tedious, you can use a function to do the exporting for you. See the Closure Library functions goog.exportSymbol() and goog.exportProperty() for examples of exporting functions.
However, this seems pretty convoluted, and makes me directly dependent on Google Closure Compiler. At this point, I am looking for info about future standards from the EcmaScript Committee, with regards to CommonJS, and for any wisdom from people who have thought about this problem and attempted to tackle it. Especially from fellow TypeScript developers; I don't want to have to create declare files in TypeScript for sugar.js, then use Google Closure Compiler against my TypeScript compiler output. It just sounds impossibly complex and hard to debug.

Unfortunately, there's nothing built into Javascript to do this "shaping", and really what you want is a compiler anyway, since roughly one of the roles it serves is to automate "shaping" at many levels (not just with methods).
The Closure Compiler is mature and open source (JQuery is actually minified using the Closure Compiler). So if you're going to begin annotating your JS code for a compiler, it might as well be this one.
There a huge number of side benefits of using a compiler, btw. They will reduce your file sizes (and thus interpretation/run time) a great deal more than simply shaping your included libraries. And while you're developing it will produce a lot of helpful messages to catch bugs early.

When is a javascript engine (like Spidermonkey) going to build in a library like jquery, mootools, etc.. and how can anyone help this to come about?

We all know the great benefits that js libraries such as jquery and mootools etc. have contributed to web browsers and web development. These libraries are now included in a lot if not most of all websites.
So, I was wondering why none of the current javascript engines just include these functionalities inside the javascript engine itself. No doubt this has even more benefits such as performance, no need for external loading, standardisation (and its own benefits), etc.
I realize that this would probably only benefit web browsers and alike, though there must be also many uses beyond just web browsers, but for the sake of argument, one could just add such engine built in functionalities in an optional engine / ECMASCript -and I am guessing the word here- component (with emphasis on optional), that could then be enabled or added only in the engines inside web browsers.
Does anyone know this or has more info on this all?
My second questions is: If we, the community, would decide this to be a great progress for the future, where can we propose/ask such a thing and what else can we do to make this happen?
(Some of you must realize the trouble that some feature cost to be included in some projects, such as the years to decade old feature requests voted on by zillion users and never got through because of ...well..let's not be ungrateful to developers and leave these dots for your own imagination. So I'd rather have that the community focus this wish on 1 place only nad maybe the answer to this second question is the start of it?)

ECMAScript only intends to standardize the minimum amount of language and support library necessary to build these higher level libraries you're describing. Also, things like jQuery work on the Document Object Model provided by the browser code, which isn't even part of the ECMAScript standard -- ECMAScript just knows about DOM nodes in the general category of "foreign objects". The SpiderMonkey engine implements just the JavaScript language and its small standard library, which is then embedded into the larger Firefox browser environment.
So, to answer the question more directly: is it possible to give the JavaScript engine intimate knowledge (and perhaps an implementation) of a user-level library like jQuery? Yes, although you'd be breaking a lot of componentization in the browser, as you mention. Will anybody actually do it? Most likely not, because JavaScript engines just implement the core of what's necessary to build higher level libraries, like jQuery. Everybody is happy with them living outside of the JS engine, and a nice property of JavaScript is that you can just load the library in as you need it -- the source is freely available.
In fact, as a further note, JS engines are doing more and more to push ECMAScript standard library code out of C++-implementation land and into something called "self-hosted builtins", which enables functions like Array.indexOf to be implemented in JavaScript itself (i.e., with a for loop and comparisons). This exposes more JavaScript code to the natural process of optimizing JIT compilers, instead of having to deal specially with calls into native C++ implementation code.

Building CoffeesSript browser applications - namespace and scope issues

I am new to CoffeeScript and am trying to get a feel for the best way of managing and building a complex application that will run in the browser. So, I am trying to determine what is the best way to structure my code and build it; with consideration for scope, testing, extensibility, clarity and performance issues.
One simple solution suggested here (https://github.com/jashkenas/coffee-script/wiki/%5BHowTo%5D-Compiling-and-Setting-Up-Build-Tools) seems to be maintain all your files/classes separately - and the use a Cakefile to concatenate all your files into a single coffee file and compile that. Seeems like this would work, in terms of making sure everything ends up in the same scope. It also seems like it makes deployment simple. And, it can be automated, which is nice. But it doesn't feel like the most elegant or extensible solutions.
Another approach seems to be this functional approach to generating namespaces (https://github.com/jashkenas/coffee-script/wiki/Easy-modules-with-CoffeeScript). This seems like a clever solution. I tested it and it works, but I wonder if there are performance or other drawbacks. It also seems like it could be combined with the above approach.
Another option seems to be assigning/exporting classes and functions to the window object. It sounds like that is a fairly standard approach, but I'm curious if this is really the best approach.
I tried using Spine, as it seems like it can address these issues, but was running into issues getting it up and running (couldn't install spine.app or hem), and I suspect it uses one or more of the above techniques anyways. I would be interested if javascriptMVC or backbone solves these issues - I would consider them as well.
Thanks for your thoughts.

Another option seems to be assigning/exporting classes and functions to the window object. It sounds like that is a fairly standard approach, but I'm curious if this is really the best approach.
I'd say it is. Looking at that wiki page's history, the guy advocating the concatenation of .coffee files before compilation was Stan Angeloff, way back in August 2010, before tools like Sprockets 2 (part of Rails 3.1) became available. It's definitely not the standard approach.
You don't want multiple .coffee files to share the same scope. That goes against the design of the language, which wraps each file in a scope wrapper for a reason. Having to attach globals to window to make them global saves you from making one of the most common mistakes JavaScripters run into.
Yes, helper duplication duplication can cause some inefficiency, and there's an open discussion on an option to disable helper output during compilation. But the impact is minor, because helpers are unlikely to account for more than a small fraction of your code.
I would be interested if javascriptMVC or backbone solves these issues
JavaScript MVC and BackBone don't do anything with respect to scoping issues, except insofar as they cause you to store data in global objects rather than as local variables. I'm not sure what you mean when you say that Spine "seems like it can address these issues"; I'd appreciate it if you'd post another, more specific question.

In case you would prefer the node.js module system, this gives you the same in the browser: https://github.com/substack/node-browserify
File foo.coffee:
module.exports = class Foo
...
File bar.coffee:
Foo = require './foo'
# do stuff

Should writing Javascript be avoided in favour of GWT/WebSharper or some other abstraction?

I'm curious what's the view on "things that compile into javascript" e.g. GWT, Script# and WebSharper and their like. These seem to be fairly niche components aimed at allowing folks to write javascript without writing javascript.
Personally I'm comfortable writing javascript (using JQuery/Prototype/ExtJS or some other such library) and view things like GWT these as needless abstractions that may end up limiting what a developer needs to accomplish or best-case providing a very long-winded workaround. In some cases you still end up writing javascript e.g. JSNI.
Worse still if you don't know what's going on under the covers you run the risk of unintended consequences. E.g. how do you know GWT is creating closures and managing namespaces correctly?
I'm curious to hear others' opinions. Is this where web programming is headed?

Should JavaScript be avoided in favor of X? By all means!
I will start with a disclaimer: my answer is very biased as I am on the WebSharper developer team. The reason I am on this team in the first place is that I found myself a complete failure in writing pure JavaScript, and then suggested to my company that we try and write a compiler from our favorite language, F#, to JavaScript.
For me, JavaScript is the portable assembly of the web, fulfilling the same role as C does in the rest of the world. It is portable, widely used, and it will stay. But I do not want to write JavaScript, no more than I want to write assembly. The reasons that I do not want to use JavaScript as a language include:
There is no static analysis, it does not even check if functions are called with the right number of arguments. Typos bite!
There is no or a very limited concept of libraries, namespaces, modules, classes, therefore every framework invents their own (a similar situation to that of R5RS Scheme).
The tooling (code editors, debuggers, profilers) is rather poor, and most of it because of (1) and (2): JavaScript is not amenable to static analysis.
There is no or a very limited standard library.
There are many rough edges and a bias to using mutation. JavaScript is a poorly designed language even in the untyped family (I prefer Scheme).
We are trying to address all of these issues in WebSharper. For example, WebSharper in Visual Studio has code completion, even when it exposes third-party JavaScript APIs, like Ext Js. But whether we have or will succeed or fail is not really the point. The point is that it is possible, and, I would hope, very desirable to address these issues.
Worse still if you don't know what's
going on under the covers you run the
risk of unintended consequences. E.g.
how do you know GWT is creating
closures and managing namespaces
correctly?
This is just about writing the compiler the right way. WebSharper, for instance, maps F# lambdas to JavaScript lambdas in a 1-1 manner (in fact, it never introduces a lambda). I would perhaps accept your argument if you mentioned that, say, WebSharper is not yet mature and tested enough and therefore you are hesitant to trust it. But then GWT has been around for a while and should produce correct code.
The bottom line is that strongly typed languages are strictly better than untyped languages - you can easily write untyped code in them if you need to, but you have the option of using the type-checker, which is the programmer's spell-checker. Why wouldn't you? A refusal to do so sounds a bit luddite to me.

Although, I don't personally favor one style over another, I don't think that abstraction from Javascript is the only benefit that these frameworks bring to the table. Surely, in abstracting the entire language, there will be things that become impossible that were previously possible, and vice-versa. The decision to go with a framework such as GWT over writing vanilla JavaScript depends on many factors.
Making this a discussion of JavaScript vs language X is fruitless as each language has its strengths and weaknesses. Instead, do an objective cost-benefit analysis on what is to be gained or lost by going with such a framework, and that can only be done by you and not the SO community unfortunately.
The issue of not knowing what goes on under the hood applies to JavaScript just as much as it does to any translated source. How many people do you think would know exactly what is going on in jQuery when they try to do a comparison such as $("p") == $("p") and get back false as a result. This is not a hypothetical situation and there are several questions on SO regarding the same. It takes time to learn a language or framework, and given sufficient time, developers could just as well understand the compiled source of these frameworks.
Another related aspect to the above question is of trust. We continuously build higher level abstractions upon lower level abstractions, and rely on the fact that the lower level stuff is supposed to work as expected. What was the last time you dug down into the compiled binary of a C++ or Java program just to ensure that it worked correctly? We don't because we trust the compiler.
Moreover, when using such a framework, there is no shame in falling back to JavaScript using JSNI, for example. It's all about solving the problem in the best possible manner with the tools at hand. There is nothing sacred about JavaScript, or Java, or C#, or Ruby, etc. They are all tools for solving problems, and while it may be a barrier for you, it might be a real time-saver and advantageous to someone else.
As for where I think web programming is headed, there are many interesting trends that I think or rather hope will succeed such as JavaScript on the server side. It solves very real problems for me at least in that we can avoid code duplication easily in a web application. Same validations, logic, etc. can be shared on the client and server sides. It also allows for writing a simple (de)serialization mechanism so RPC or RMI communication becomes possible very easily. I mean it would be really nice to be able to write:
account.debit(200);
on the client side, instead of:
$.ajax({
url: "/account",
data: { amount: 200 },
success: function(data) {
..
}
error: function() {
..
}
});
Finally, it's great that we have all this diversity in frameworks and solutions for building web applications as the next generation of solutions can learn from the failures of each and focus on their successes to build even better, faster, and more awesome tools.

I have three big practical issues I have with websharper and other compilers that claim to avoid the pain of Javascript.
If you won’t know Javascript well you can’t understand most of the examples on the web of using the DOM/ExtJs etc., so you have to learn Javascript whatever. (For the some reason all F# programmers must be able to at least read C# or VB.NET otherwise they cannot access most information about the .net framework)
On any web project you need a few web experts that know the DOM and CSS inside out; would such a person be willing to work with F# rather than Javascript?
Being tied into the provider of the compiler, will they be about in 5 years’ time; I want full open source or the tools to be supported by Microsoft.
The 4 big positives I see with these frameworks are:
Shareing code between the server/client
Having fewer languages a programmer needs to know (javascript is a real pain as it looks like Jave/C# but is not anything like them)
The average quality of a F# programmer is a lot better than a jscript programmer.

My opinion for what it's worth is that every framework has its pros/cons and a project team should evaluate their use cases before including one. To me any framework is just a tool to be used to solve a problem, and you should pick the best one for the job.
I prefer to stick to pure JavaScript solutions myself, but that being said I can think of a few cases where GWT would be helpful.
GWT would allow a team to share code between the server/client, reducing the need to write the same code twice (JS and Java). Or if someone was porting a Java client to a web UI, they may find it easier to stick to GWT ( of course then again it may make it harder :-) ).

I know this is a gross over-simplification, because there are many other things that frameworks like GWT offer, but here is how I view it: if you like JavaScript, write JavaScript; if you don't, use GWT or Cappuccino or whatever.
The reason people use frameworks like GWT is not necessarily the abstraction that they give--you can have that with JavaScript frameworks like ExtJS--but rather the fact that they allow you to write web applications in something other than JavaScript. If I were a Java programmer who wanted to write a web application, I would use GWT because I would not have to learn a new language.
It's all preference, really. I prefer to write JavaScript, but many people don't.