We Keep Coding

Javascript: How to make sense of all the frameworks and design philosophies?

I have been a user of jQuery (and some of its minor plugins) for a while. The Javascript code I've developed over the years could be described best as... messy. It used a ton of global variables and functions here and there, didn't use standard ways of organizing the code, nor any design patterns whatsoever.
I am currently building the new version of a website, and I have completed doing the backend with PEAR::MDB2 and Smarty templates. The rest is just homebrew PHP with some classes.
Now I am at the point where I'll add the Javascript layer on top of the website to improve the user-friendliness of some features. (while making sure everything degrades gracefully) I want to write better, cleaner, more organized Javascript than I used to, so I did a little research. I read Stefanov's Object-Oriented Javascript to have a better grasp on some concepts I knew only loosely about (prototypes, constructors, etc.) as well. Now I'm stuck at a point where I wonder which Javascript frameworks I should use, and how to organize it all.
After conducting my research, I understood Cappuccino & Objective-J, and Sproutcore were not what I was looking for. To quote Cappucino's about page:
Cappuccino is not designed for building web sites, or making existing sites more "dynamic". We think these goals are too far removed from those of application development to be served well by a single framework. Projects like Prototype and jQuery are excellent at those tasks
So there's that. Then I found out about Coffee Script, which is more of a one-to-one "compiler" and wouldn't help me with the actual organization of my code.
I also stumbled on some articles that give guidelines:
Using Inheritance Patterns to Organize Large jQuery Applications
A JavaScript Module Pattern
I also found out about Backbone.js, Shoestring, JavaScriptMVC, Google Loader, jQuery Tools, jQuery UI. I don't really know what to do of all this... The things I know:
I don't want to invest too much time in learning something too complex, I want to keep things simple and flexible as much as possible (that is why I don't use Symfony on the backend, for example), yet clean and organized.
I want to use jQuery, the question is, what should I use with it? (that is compatible too)
Right now, I'd use jQuery and jQuery Tools and "organize" all that in a simple namespace/object literal with simple properties and methods and also, since the site is localized, I just plan on using the simple vsprintf (as I do on the backend) with key:value pairs loaded from an object literal provided by the backend. JavaScriptMVC seems interesting, but I fear it would bring way too much complexity for a project that is fairly small sized. That is where I need your advice! Thank you very much in advance.

Ok, my attempt at an answer:
There is no 'best' to way to do it. You now know what's there and I think you might have a preference for yourself for what you want. In that case, pick a framework and learn it inside-out. (sorry to burst your bubble, but each framework has a learning curve, some steep, some very easy, but in the end to use it well you have to invest in it. Just do it, you won't be sorry).
You of course have an preference for clean code, so you might take some considerations into account. You also say you have a preference for jQuery, which is fine, but there are some limitations (as also pointed out in the link provided by eskimoblood).
There are some nice lectures / and tutorials with advice on how to structure your code in jQuery:
How to manage large jquery apps
On Large jQuery apps
Essential Javascript and jQuery patterns (free ebook)
Some style guides:
Jquery core UI Styleguide
Google Closure Javascript Style Guide
Tools for checking your code
JSLint
JSHint (a more forgiving/practical fork)
Closure Linter (haven't tried it yet, but intend to)
Standard works (javascript)
Everything by Douglas Crockford
Quirksmode
There might be more.. perhaps more people can contribute, but I also think that you've almost reached the end of what you can learn before getting your hands dirty. Many of these guides are written in a very generic way, but the interesting thing is that javascript is called upon in many specific situations. It might be useful to just post some of the code that you regard as "messy" and we can help you figure out how to do it better. Good luck!

You should watch the video and read the links in this article and then you should ask yourself again if jquery is the right tool. Maybe you will use dojo, that is much better for larger projects or you take a look at backbone and where you can stay with jquery. After all both of them are more "javascriptish" then something like sproutcore, cappuciono or even GWT. And also much easier to understand when you come from jquery.

One framework that is to consider is definitely ReactJS from Facebook. This framework is pretty slick in many ways.
First thing you have to know is that it is a view framework. It can be used server-side to do the pre-rendering of pages, but it really shines on client side. Since it's a view framework, it can be used with backbone or any other "back-front"-end framework.
One of the main point of React is its rapidity. It keeps a virtual DOM in memory and virtualize all the webpages events. So the virtuals event are used to keep events browser agnostics.
The virtual DOM kind of make programming a dynamic site as if you were programming an old static website. You can just shoot the whole HTML to render to the view engine (as if you were "re-rendering" the whole page) and it will manage the DOM operations. It does a diff between the new virtual DOM and the current virtual DOM and only inserts nodes that needs to be inserted. This way you reduce the number of DOM ops and thus increase your render speed by a lot.
A good place to start is this tutorial which shows how to use "Flux" (the web flow designed by Facebook for its site) in order to realize a Todo application!

I'm interested in using Ext.js, but troubled by a number of issues. Is there a better alternative?

Ext.js is of interest to me because it appears to have a fairly complete widget set (though I've certainly run into shortcomings of the widgets after a very short time). The problems I potentially have with it, however, include these:
in my opinion it's very ugly, and looks like a cross between a Windows UI and some PHP content management system (yes I know I can write my own theme, but there may be limits to what I can change, and it may not be as easy as it should be, and I'd rather work with something that is clean, elegant, and attractive out of the box.)
the graphs currently use Flash, which limits the clients it can be useful on (version 4 promises to resolve this problem)
the license is a bit restrictive, which is not an issue now, but it makes me think twice about becoming intimately familiar with a tool that I may not be able to use in the future.
Honestly the look & feel is probably the biggest issue to me: I've seen in the past that developers (or companies) who don't place a high value on aesthetics often can't be bothered by other 'details' either.
So: is there a other better alternative?
UPDATE:
Another problem with Ext.js, is that it seems to be an all-or-nothing proposition. That's one reason I'm not looking seriously at SproutCore, which in other respects is awesome. (Well... it needs some performance improvements, but it'll get there eventually I think...)
Cappuccino seems to be the same way: you're not using it on a web page, rather you're building a Cappuccino app that just happens to use the web as its runtime environment.
In other words, ideally I don't want a JS version of Flex: it's own little walled garden. jQueryUI would be ideal if it were more complete, since it doesn't break with normal web development methods and paradigms. But if walled gardens are all there are, I'll live with it.

If you are looking for a RIA-Framework you shouldn't use jQuery(UI). Instead use one of the following frameworks.
Qooxdoo (www.qooxdoo.org): Pure Open-Source RIA-Framework with the most of the features of ExtJS. There were coming up some interessting new themes in the last weeks
Cappuccino (http://cappuccino.org): Nice framework with a nice GUI
ZK (http://www.zkoss.org/): ExtJS like RIA-Framework
SproutCore (http://www.sproutcore.com/): Apple-Style RIA-Framework

I undestand your pain - I use to got same - in ExtJS I saw only problems (those what you mentioned, and belive me much more others), however because I was a little forced to use it, I started to use it and I followed in love - It very complex and optimal solution for javascript driven apps.
You can go and look for alternatives like jQuery (I was there for 1 year), dojo, and others frameworks, but whoever says that they are better than ExtJS, he rather did not have ..... time to know ExtJS
It my very subjective opinion

I dont know much about PHP, and I am using ExtJS with ASP.Net MVC, they fit well. I wish I could use an integrated JS Widget set, but Jquery UI widget set was not rich enough when I last evaluated and I dont think it still is. But it is at least included in the last version of MVC.
My only complain about ExtJS would be the number of user extension controls that you need to use, I'd like to see those as part of ExtJS core widget set, but they are not, but still distributed with the library. ExtJS is a very nice fit for a cross-platform Rich Web Application. I would recommend trying ExtJS, I have really no complains about it other that what I already said above.

How to start developing Javascript apps?

I started out with Rails development, but soon I realized Rails without JS is pretty much useless. So, I am trying to play with JS in my free time.
So, what are the "tools of trade (IDEs if any)" for JS development?
My primary focus is usage of APIs, Ajax etc, so that I don't get lost when I get develop Rails apps which uses JS.
There are many tutorials for using JS for Open graph API or twitter API, but that is to get things done and abstract some feature sets which a beginner should know.
I generally start learning a language by making a calculator, end to end (always works for me). But I cannot make a calculator and use API/Ajax calls.
So, what are your suggestions?
PS: I am aware about Douglas Crockford's video lectures, they are awesome, but I need some thing more concrete.
UPDATE:
My 2 original questions:
1. What are the tools of trade for JS? Like Eclipse:Java::X:Javascript, What is X (multiple Xs are allowed)?
2. What sample app do you recommend for me to start with?

I think the most important thing is: Know your language!
JavaScript libraries are useful, but if your do not know the language, you have no chance of getting something done that cannot be achieved using your library of choice. For example: JavaScript's prototype-system is very valuable and important for building robust applications. I really recommend that you experiment with this before diving into something you barely know. Closures are also very important to understand.
One thing is important, though: Do not spend too much time on browser incompatibilities. This is what most libraries are for. What you need to know is how the language works. Implementation-specific things are not worth learning (in most cases), since somebody already fixed those for you.
#PS: Douglas Crockford indeed does a pretty awesome job on explaining how JavaScript really works. You should keep watching this.
#UPDATE:
I think the X has it's origin in JavaScript's MIME-type: application/x-javascript. My tool of choice for web-development (including JavaScript) is NetBeans IDE. I use it in my workplace and it is very reliable and comfortable to work with. This is subjective (of course). Most other IDE's will do, too.
This is difficult to answer. I will update my post if something comes to my mind.

(I realize my comments below typically refer to using javascript with regards to websites, so ymmv if you're using js for a different environment, like couchDB)
For an ide, any text editor will do, but I prefer convenient features like:
syntax coloring (because it's so pretty)
(s)ftp plugins - makes saving files convenient
I would probably go with something like notepad++, or aptana studio (although I find aptana to be a bit of overkill at times, but very powerful). Adobe Dreamweaver works nicely too, if you wanna pay. Otherwise, the bare minimum would be notepad and an ftp client (again, with regards to front-end web development work)
For debugging or trying out quick javascript, I find safari's and chrome's built-in development tools handy, but on firefox, it's firebug all the way. Firebug is awesome.
I notice that in my work environment that some people (particularly the business side) use the words javascript and the DOM interchangably, but I find that their attempts at explaining something simply leads to confusion. That being said, if you're writing web apps, I find the mozilla DOM reference to be pretty awesome and used to hit it up all the time before I got familiar: https://developer.mozilla.org/en/DOM
Actually developer.mozilla.org is overall pretty awesome.
Other than that, getting started is simply a matter of learning the syntax. I would probably get used to the syntax first (which should be pretty quick anyways) before I start thinking about other concepts like learning how to prototype and whatnot (which you can look up on this site).
A framework like jQuery is very convenient for various repetitive tasks, but I wouldn't necessarily jump straight into using jQuery without being familiar with the js syntax first.

The most important thing you should learn is jQuery. It is now the de facto standard javascript library.
jQuery is great regardless of what you are doing with it, but since you mentioned AJAX, I'll point out that jQuery makes AJAX stupid easy, because you can do things like this:
$.ajax({
url: 'some/ajax/endpoint',
success: function(data) {
// do something with data
}
});
Obviously there are a lot of other options you can pass along, but the basic structure is really simple and easy to use.
jQuery also has a really powerful syntax for selecting parts of the DOM and adding events. For example, if you wanted to catch anytime some clicked on an image and tell them the image src, you can do something like this:
$('img').click(function() {
alert($(this).attr('src'));
});

Best Technologies for AJAX Web Development

I have some experience in AJAX development, mostly on .NET and MooTools. However, I want to learn more and see what others out there thought about the various other options available. I am looking more for advice about the front end. The back end, I will most probably be coding it in .NET using c# and WCF services.
Please feel free to provide me as much information as you can. Also, I would appreciate any links to resources.
List of Options (feel free to add)
Write my own Javascript
Use a framework like MooTools, JQuery, etc. Which one is better?
Use Google Web Toolkit. Am I tying myself to the limitations of GWT? Or are there not limitations?
ASP.NET AJAX
WPF (Will this run on non-IE browsers?)
Flash (it'll be a pain to learn action script)
Thanks
Jaspreet

Writing your own Javascript often means reinventing the wheel when it comes to trying to attain cross-browser compatibility. Frameworks such as jQuery and MooTools take care of much of that grunt work for you.
Personally, I would suggest picking a Javascript framework or using GWT, but really it's just a matter of personal preference at that point, much like choosing a programming language. Pick that one that is best for your purposes, or that you're most familiar with.

I'd go with jQuery.
jQuery will be bundled with the next version of Visual Studio. Google uses jQuery. jQuery has the largest user-base of any of the frameworks out there.. And speaking of not re-inventing the wheel: jQuery also has the largest plugin-repository of any of the frameworks out there.. Including its own official UI library which is in constant development.
.. And if you're a fan of books; there's 3-4 books about the framework out currently.
Oh, yeah! Check out Douglas Crockford's website for great vanilla JavaScript tips and tricks.
Just my two cents :)

jQuery
prototype
MooTools
dojo
GWT
ExtJS
Those will all have you working with very solid code, which is usually the best way to improve your own skills. And don't forget all their extensions/plugins.

I would recommend jQuery. It's more extensible and light-weight than most other JavaScript libraries I've seen (and you can have it Google-cached with the AJAX APIs).
For Flash-based AJAX web apps, Flex is a better solution.

Since this is a fairly subjective question I will put in my 2 cents.
I've developed using the Dojo toolkit and found it very well rounded. However, if you don't do a custom build on your library it slows everything down. It has a steeper learning curve than other frameworks, but seems to sit pretty close to heart of Javascript.
But this isn't an advertisement for dojo. I'm working on a Ajax web app currently and my goal is to try and avoid using a framework. Why? A few reasons.
The biggest reason is complete control over the code. Not that I don't trust toolkits, I just like my custom code. I don't have any code that I don't use, and all of my abstractions are custom made for my purposes. At times I am in fact reinventing the wheel*, but its always a bit different because it's purpose built.
The other big reason is the amount of knowledge I gain. I'm confident that I could pick up any framework much faster, because of not using one for a project. I understand so much more than I did previously. Before It was more like a voodoo mystery box, and now even frameworks make more sense to me.
The fact of the matter is if you do any non-trivial javascript you will end up with a framework (or a mess). In reality it may not be the fastest way to write web apps, but it isn't all that hard with a little practice. You can be amazed at what creative things you can come up with that didn't exist before.
*Reinventing the wheel is a terrible analogy that we all use. We reinvent wheels all the time. First made of stone, then lighter wood, tires, etc... My truck doesn't have the same wheels as my sedan.

Like other responders I prefer to use frameworks, but my favorite is dojo -- clean and well-designed architecture, good books (my favorite being Matthew Russel's), and a particularly impressive approach to using browser extensions (such as Google Gears or Microsoft Silverlight) if the user has them installed, while falling back gracefully (e.g. to Ajax-interfaced server-side storage) if no suitable browser extensions (or sufficiently advanced browsers, with HTML5 support) are present.
Dojo is, currently, slightly less popular than jquery (e.g., [jquery ajax] has 8.4 million hits on Google Search, vs 4.3 million for [dojo ajax]), but that's still way popular enough to ensure it won't go away any time soon;-).

You must check ExtJS.
Most libraries I've seen have usable core but lack the widget/control library that will blend-in with each other. Sure you can get a vast number of controls for jQuery, but do you have the time to find out which ones have intended functionality and tweak their visual appearance to match the rest of controls? ExtJS gives you just that. Core is core. It works and it works great. But the vast selection of compatible and functionality rich controls is the main driver that will make your life easier.
I've used ExtJS with Asp.Net and WCF services and it was a really nice experience overall. You will need some time to get to know ExtJS component model to extend it even further (as you would need to do the same with other libs), but when you do, you start loving it even more.
One more thing: since MS ships jQuery with MVC it doesn't mean it's the best. It only means it's one of the best (and much better than Microsoft's in-house Ajax) but with the most relaxed open source license.
Edit: Now that Silverlight 3 has shipped it became apparent it will shake things a bit with its offline support and rich graphics support. As well as standalone desktop like web apps. Definitely something to consider...

Please note that WPF has nothing to do with ajax.
AJAX = Asynchronous javascript and xml
WPF = a windows-only UI display technology. The only option there for in-browser use is to create an Xbap. The more-suited-for-the-web counterpart is silverlight (say, like flash but from microsoft), so perhaps you should look into that as well... it's like using flash, but with familiar MS technologies.
Concerning your question about which framework to use in case of ajax: I prefer jQuery. It's short syntax and chaining allow you to reach your goals very fast in an intuitive way.
Also, GWT is a Java framework that also renders stuff on the client, as far as i know. So you would have the limitation of Java as backend language...

GWT has made considerable leaps and bounds over the past 7 months. The mutiple GWT widget libraries have received a good deal of attention, and Google's work on the library has been impressive. Going GWT limits you to:
Using GWT widgets
Possibly becoming a GWT widget developer
The benefits include:
Coding in Java
Debugging in Java
Skipping the parts of JavaScript that make you pull your hair out
GWT is the step past AJAX. It buries the HTML / JavaScript so that you don't need to deal with it, but also provides access in case you do want to deal with it.
Overall, you will spend less time trying to get small things working, and more time adding awesome to your application.

there is also prototype.js

Try the Yahoo! User Interface Library (YUI)

Pick a library to learn, then pick up the others--but I don't think I'd reinvent the wheel.
I picked MooTools and have picked up jQuery/Prototype since. (Picked Mootools because their site, at the time, was the only one that validated, figured clean html meant clean javascript).
Side note about MooTools:
An often overlooked feature of MooTools is the download builder. You can download just enough of the framework source to get your AJAX working, and nothing else.

There's the BBC's open source javascript library called Glow

Should I use Google Web Toolkit for my new webapp?

I would like to create a database backed interactive AJAX webapp which has a custom (specific kind of events, editing) calendaring system. This would involve quite a lot of JavaScript and AJAX, and I thought about Google Web Toolkit for the interface and Ruby on Rails for server side.
Is Google Web Toolkit reliable and good? What hidden risks might be if I choose Google Web Toolkit? Can one easily combine it with Ruby on Rails on server side? Or should I try to use directly a JavaScript library like jQuery?
I have no experience in web development except some HTML, but I am an experienced programmer (c++, java, c#), and I would like to use only free tools for this project.

RoR is actually one of the things the GWT is made to work well with, as long as you're using REST properly. It's in the Google Web Toolkit Applications book, and you can see a demo from the book using this kind of idea here. That's not to say that you won't have any problems, but I think the support is definitely out there for it.
There's a neat project for making RoR/GWT easy that you can find here (MIT license). I haven't had a chance to try it out yet, but it looks like a good amount of thought has been put into it. One catch is that it looks like it hasn't been fully tested with 2.1 Rails yet, just 2.0, so you may run into a few (probably minor and fixable) errors.

If you are looking to integrate GWT with non-Java backends such as ROR, PHP etc., you should bear in mind that GWT 1.5 now supports JavaScript Overlay types. This feature lets you write classes that can be mapped over the top of native JavaScript objects to easily provide accessor methods for properties of those objects and other extended functionality.
See this link for more details:
JavaScript Overlay Types
So you could return JSON encoded data from your backend via AJAX calls, parse it into a JavaScript Object and then access the data through your GWT Java code using the overlay classes you've created. Or when you render your page you can render static config data as JavaScript Objects and read it in via this mechanism, rather than having to do an AJAX call to grab the data.

If you know JAVA, and have somewhere you can host it (like a tomcat or glassfish container) I would recommend that much more than using Ruby for the back end. The main reason is that then you can share all of your objects, and use the built in RPC mechanism. I've done this for quite a lot of our projects and it's a huge timesaver, not to mention that the code is less error prone, because you don't convert your java objects to anything and then back again.
I have linked my GWT with Rails before, using the to_json function in Rails and then reading the JSON in GWT. It's all supported, but it is far more annoying than just doing the back end in JAVA.
Of course if you have cheap hosting, then Java containers are pretty much out of the question, in which case I would think Rails would be the next best thing.

GWT is very high quality with a great community. However you do need to know CSS if you want to adjust the look of things (you will) - CSS can do a lot of the layout, just like regular web if you want it to. Libraries like GWT-ext or ExtGWT can help a bit as they have stunning "out of the box" looks but for a price (extra size to your app).

You can code everything in Java using GWT, and you can integrate existing 3rd party javascript libraries with it. It's very good. I've never used RoR much though, so can't say anything about that.

If you're experienced in Java but not in Javascript/CSS, then GWT is going to be a lifesaver (unless you want to learn them, of course). CSS has so many little fiddly details. It is not uncommon to spend half a day fixing a 2 pixel misalignment that only occurs in IE6.
I am not sure about how easy it would be to use ROR for the back end... It is possible, I am sure, since GWT ajax communication is just servlets. But they provide some really nice functionality for passing Java objects back and forth which you won't be able to utilize if your server isn't also using Java.

I wrote about some of the disadvantages of GWT recently. Mainly, the disadvantages are: long deployment cycle for changes to some parts of the application and a rather steep learning curve. As a seasoned Java programmer, the second should be less of a problem and if you use a seperate backend, the first is also mitigated (as a complete redeploy is primarily required when you change the 'server' part of the application).

GWT is a wonderful framework with lots of potential. Keep in mind that it's still quite new, though. There are some unresolved bugs that can really annoy you, and they usually require ugly workarounds to get past. The community is great but you'll probably end up with a few problems sooner or later that Google can't answer yet.
But hey, I say go for it. The potential for GWT is awesome, and I bet it's future will be bright.

You should definitely use GWT for a new project (it's pretty easy to use in an old project too).
I my experience it's very fast to learn and use. The compiled javascript code is much better than anything you could ever write by hand and it works fast too.
Another benefit is the ability to debug you're code (which is hell with javascript alone)

This blog has inputs from many experienced users of GWT and have some great discussion points. I personally have huge experience with varied UI Frameworks. I will add my two cents. Lets look at fundamental advantages and disadvantages of GWT
Fundamental Advantage
GWT takes the web layer programming to JAVA. So, the obvious advantages of Java start getting into play. It will provide Object Oriented programming. It will also provide great debugging and compile time checks. Since it generates HTML and Javascript, it will also have ability to hide some complexity within its generator.
Fundamental Disadvantage
The disadvantage starts from the same statement. GWT takes the web layer programming to JAVA. If you know JAVA, probably you will never look out for an alternative language to write your business logic. It's self sufficient and great. But when it comes to writing configurations for a JAVA application. We use property files, database, XML etc. We never store configurations in a JAVA class file. Think hard, why is that?
This is because configuration is a static data. It often require hierarchy. It is supposed to be readable. It never requires compilation. It doesn't require knowledge of JAVA programming language. In short, it is a different ball game. Now the question is, how it relates to our discussion?
Now, lets think about a web page. Do you think when we write a web page we write a business logic? Absolutely not. Web page is just a configuration. It is a configuration of hierarchical containers and fields. We need to write business logic for the data that will be captured from and displayed on the web page and not to create the web page itself.
Previous paragraph makes a very very strong statement. This will explain why HTML and XML based web pages are still the most popular ones. XML is the best in business to write configurations. A framework must allow a clear separation of web page from business logic (the goal of MVC framework). By doing this a web designer will be able to apply his skills of visualization and artistry to create brilliant looking web pages just by configuring XMLs and without being bothered about the intricacies of a programming language. Developers will be able to use their best in business JAVA for writing business logic.
Finally, lets talk about the repercussions in direct terms. GWT breaks this principal so it is bound to fail. The cost for developing GWT application will be very high because you will need multiskill programmers to write web pages. The required look and feel will be very hard to achieve. The turn around time of modifying the web page will be very high because of unnecessary compilation. And lastly, since you are writing web pages in JAVA it is very easy to corrupt it with business logic. Unknowingly you will introduce complexities that must be avoided.

You could also consider Grails ("Groovy on Rails") which gives you the benefits of a Rails framework and the use of the Java VM.

Our team recently asked the same question, and we chose to go with GWT, especially since the designer plugin made working with GWT more accessible to non-java experts on the team. Whoever makes this choice, just beware you DON'T use the GWT Designer plugin !! It has not been updated (in at least a year, apparently) to create a GWT application that is compatible with IE8.
Our team had almost completed our application layouts, which were working perfectly in Chrome, FF and Safari. Then they blew up in IE. IE 7 would load partial pages (but not composite includes), and IE8 was not even able to load up the application. It just hung.
The designer plugin has buttons that allow the user to add CellTable widgets that are not IE compatible (CellTable, DeckPanel, Horizontal Panel, Vertical Panel, among others). These will cause intense pain when the layouts have to be re-done in java without assistance from the designer.
Experienced GWT users love it, but the designer plugin will kill you.