We Keep Coding

Javascript: How to make sense of all the frameworks and design philosophies?

I have been a user of jQuery (and some of its minor plugins) for a while. The Javascript code I've developed over the years could be described best as... messy. It used a ton of global variables and functions here and there, didn't use standard ways of organizing the code, nor any design patterns whatsoever.
I am currently building the new version of a website, and I have completed doing the backend with PEAR::MDB2 and Smarty templates. The rest is just homebrew PHP with some classes.
Now I am at the point where I'll add the Javascript layer on top of the website to improve the user-friendliness of some features. (while making sure everything degrades gracefully) I want to write better, cleaner, more organized Javascript than I used to, so I did a little research. I read Stefanov's Object-Oriented Javascript to have a better grasp on some concepts I knew only loosely about (prototypes, constructors, etc.) as well. Now I'm stuck at a point where I wonder which Javascript frameworks I should use, and how to organize it all.
After conducting my research, I understood Cappuccino & Objective-J, and Sproutcore were not what I was looking for. To quote Cappucino's about page:
Cappuccino is not designed for building web sites, or making existing sites more "dynamic". We think these goals are too far removed from those of application development to be served well by a single framework. Projects like Prototype and jQuery are excellent at those tasks
So there's that. Then I found out about Coffee Script, which is more of a one-to-one "compiler" and wouldn't help me with the actual organization of my code.
I also stumbled on some articles that give guidelines:
Using Inheritance Patterns to Organize Large jQuery Applications
A JavaScript Module Pattern
I also found out about Backbone.js, Shoestring, JavaScriptMVC, Google Loader, jQuery Tools, jQuery UI. I don't really know what to do of all this... The things I know:
I don't want to invest too much time in learning something too complex, I want to keep things simple and flexible as much as possible (that is why I don't use Symfony on the backend, for example), yet clean and organized.
I want to use jQuery, the question is, what should I use with it? (that is compatible too)
Right now, I'd use jQuery and jQuery Tools and "organize" all that in a simple namespace/object literal with simple properties and methods and also, since the site is localized, I just plan on using the simple vsprintf (as I do on the backend) with key:value pairs loaded from an object literal provided by the backend. JavaScriptMVC seems interesting, but I fear it would bring way too much complexity for a project that is fairly small sized. That is where I need your advice! Thank you very much in advance.

Ok, my attempt at an answer:
There is no 'best' to way to do it. You now know what's there and I think you might have a preference for yourself for what you want. In that case, pick a framework and learn it inside-out. (sorry to burst your bubble, but each framework has a learning curve, some steep, some very easy, but in the end to use it well you have to invest in it. Just do it, you won't be sorry).
You of course have an preference for clean code, so you might take some considerations into account. You also say you have a preference for jQuery, which is fine, but there are some limitations (as also pointed out in the link provided by eskimoblood).
There are some nice lectures / and tutorials with advice on how to structure your code in jQuery:
How to manage large jquery apps
On Large jQuery apps
Essential Javascript and jQuery patterns (free ebook)
Some style guides:
Jquery core UI Styleguide
Google Closure Javascript Style Guide
Tools for checking your code
JSLint
JSHint (a more forgiving/practical fork)
Closure Linter (haven't tried it yet, but intend to)
Standard works (javascript)
Everything by Douglas Crockford
Quirksmode
There might be more.. perhaps more people can contribute, but I also think that you've almost reached the end of what you can learn before getting your hands dirty. Many of these guides are written in a very generic way, but the interesting thing is that javascript is called upon in many specific situations. It might be useful to just post some of the code that you regard as "messy" and we can help you figure out how to do it better. Good luck!

You should watch the video and read the links in this article and then you should ask yourself again if jquery is the right tool. Maybe you will use dojo, that is much better for larger projects or you take a look at backbone and where you can stay with jquery. After all both of them are more "javascriptish" then something like sproutcore, cappuciono or even GWT. And also much easier to understand when you come from jquery.

One framework that is to consider is definitely ReactJS from Facebook. This framework is pretty slick in many ways.
First thing you have to know is that it is a view framework. It can be used server-side to do the pre-rendering of pages, but it really shines on client side. Since it's a view framework, it can be used with backbone or any other "back-front"-end framework.
One of the main point of React is its rapidity. It keeps a virtual DOM in memory and virtualize all the webpages events. So the virtuals event are used to keep events browser agnostics.
The virtual DOM kind of make programming a dynamic site as if you were programming an old static website. You can just shoot the whole HTML to render to the view engine (as if you were "re-rendering" the whole page) and it will manage the DOM operations. It does a diff between the new virtual DOM and the current virtual DOM and only inserts nodes that needs to be inserted. This way you reduce the number of DOM ops and thus increase your render speed by a lot.
A good place to start is this tutorial which shows how to use "Flux" (the web flow designed by Facebook for its site) in order to realize a Todo application!

Need help determining how to approach building my project

I'd like to create something similar to a family tree online app (like geni.com). I'm unsure what languages I should use to build it with. My IT strong points aren't in programming and this project is going to require me to sit down and learn some languages. My problem right now is that I don't know what languages I should use.
So with the idea of a family tree online app in mind here are some of the specifications.
- I do not want to use flash.
- The app needs to be zoomable and scrollable (sort of like google maps)
- The app needs to be able to add content without reloading the page. Perhaps there's a little "+" sign and when I click it, I can add a tag/title/description/picture
- The app needs to be able to save your work for that user to retrieve later on.
- The layout that a user is able to create in is sort of widget based where the user can add a new bubble and then in that bubble they are able to add text or content.
I started programming this with HTML5 canvas and Javascript, but I'm stuck on creating a connection to the database that isn't directly from Javascript (because that seems very insecure to me). But I'm not just stumped on how to interact securely with the database (I don't even have a database picked out), but also I'm concerned that I won't be able to build out the app with just javascript and may need something else like ajax or something but I'm unfamiliar with what each language does nowadays.

If you are starting from scratch, then the best language to use is the one you are most comfortable with. Alternately, if you don't plan to be developing the whole thing yourself and you already have some other interested parties on board then the best language to use is the one that the majority of you are comfortable with. If it's just you and you do not yet have any favorites, then look around and play with a few - it's the only way to find out if you will actually like / be effective with them.
That being said, a few of the more likely candidates these days are:
JavaScript: Long gone are the days when this language was simply a way to put the D in DHTML. These days JavaScript is a viable client and server-side language. (Others here have already recommended Node.js -- I'd also recommend NPM (node package manager) to handle your dependencies). With a little bit of planning you can reuse most of your application code on both the client and the server side. On the downside, most of the server side technology is very new (only a few years old at most) and so you may find yourself implementing tools for use in your application rather than your application itself. Finding servers that support it will also be harder, again on account of it's age.
Perl: At the opposite end of the spectrum of age, we find Perl - the first commonly deployed language used to make web applications it still powers a great variety of useful websites out there (include new ones such as Pinboard.) The tools that are popular on CPAN have been vetted under fire. The good news is that it is not going anywhere anytime soon. The bad news is, you might have to search a little harder to find a module that supports [that newest, baddest thing that just came out yesterday].
PHP: The BASIC (or Perl, depending on who you ask) of the modern web, PHP was designed from the ground up to do one thing - make building dynamic web pages easier. Its popularity means that there is quite a lot of server support (PHP + Apache + MySQL is the Model T Ford of web servers -- everyone can afford one) and an enormous amount of pre-built code available for perusal. However, like BASIC, PHP's strength is also its greatest weakness. Almost anyone can write something that works in PHP ... how well it works depends on who wrote it. The caveat emptor that applies to all code snippets found on the web applies in spades to snippets of code written in PHP.
Python: The language that made programing fun again (at least for those who can see past the significant whitespace and lack of blocks / anonymous functions and overlook < 3.x's issues with non-ASCII out of the box.) It's a general-purpose, flexible and multi-paradigm language with quite a substantial standard library (but without .NET or Java's incredible bloat). In addition, quite a large amount of work has been done in it, so there is a good chance that what you need has been already developed by somebody else. Plus, it can make you fly.
Ruby (with or without Rails): The language that made the web fun, coupled, if you so desire, with the framework that made MVC cool. There is lots of documentation out there, and a great community, with many prebuilt tools (called gems) from which to pick and choose - free and cheap servers are not as common as their PHP counterparts, but they are likely to be of higher quality (when chosen at random).
All that being said, they are all great languages for web development. What matters is not what we think you should use ... but what you are most likely to be effective with. All of the languages listed above are mainstream (or will be in the next two years), easy-to-learn and easy-to-write languages. You cannot go wrong, no matter what you choose to start off with.
Alternately, if you want something a little more difficult, or less mainstream ... I am working with .NET applications at work, and with Lisp (SBCL)-based services in my spare time. I have heard great things about Lua and Java too ... there are at least two C++ web frameworks out there ... and I'm sure that there is somebody is having fun building a web service in COBOL with a FORTRAN backend. ;-)

As someone has already pointed out, you will need to work with a server side language as well. (Ruby, Python, PHP) You are exactly right there you should not be attempting a database connection via javascript in the browser.
You'll need to build out a server side application to handle the basic operations of your application.
I'd strongly recommend reading up on the MVC design pattern, and possibly looking into Ruby on Rails as your backend framework, it plays very nicely with ajax like features, and has a somewhat shorter learning curve, I believe, than some other frameworks / languages.

You will need server side scripts in a language like PHP or Ruby on Rails to interact with a database.

If you're already familiar with HTML5 and JavaScript, may I recommend using Node.JS? It's about the closest you'll get to what you already know with browser development. It can also hook in with database systems which are closer to the HTML5-suggested IndexedDB.
with that in mind...
If you're inexperienced with programming and programming languages, then the app you describe will involve a pretty big learning curve. While Flash and Flex have really nice interfaces to build apps with click-and-drag, the tools for HTML5 are much less mature.
That's not saying it's not possible with HTML5. Just that there's still some time to wait before people create tools to bring the app building process closer to what Adobe provides.

Writing a single page client-side only webpage

I'm considering to try out an idea, mostly for fun, and my question is if this is reasonable and if there are any libraries or frameworks that could make this experiment a little easier.
So, the idea: Basically it is to write a new UI for a website I've developed, but doing it with client-side code only. I can read/write data using ajax, since my existing website has an API that allows me to perform all kinds of queries. This allows me to use JavaScript for the whole thing and theoretically put all of the code in a single file.
Obviously there are limitations to circumvent; bookmarking, page refreshing, the back-button etc. But these limitations are what makes it interesting, right? :) I'm not so worried about search engine indexing, since one has to be logged in to use the site anyway.
The site itself is not overly complex, but it is not simple either. There are four different levels of users, multiple languages and quite a lot of data to be presented.
Is this a bad idea? If so, why would you advice against it? And do you know of any JavaScript frameworks or libraries that could make this easier? (And no, I'm not looking for an abstraction like Google Web Toolkit; I would like something purely JavaScript)

One of my coworkers did this. A nice feature of this concept was that you don't have a ton of POSTS whenever the user 'changes pages', since they are actually not ever changing a page until they submit their data for the final time. He did this for product registration software, which was nice. Our servers were only taking a hit when the user initially requested the page, and then when they submitted it.
The major, MAJOR downside to this concept is that most web developers are not expecting this. My coworker (and you) have a cool idea - but unless it is well implemented, with comments, 100% valid HTML, and a host of other good design principles in place, it can be confusing since most web developers have basically never seen this done before. His site was a nightmare to work with, as he did not actually know what engineering web software meant, and it was all slapped together. My organization never pursued this (potentially useful) idea because his implementation was so poor.
So, when I looked at this idea here were the trade-off I came up with:
1.) You cannot require any server-side interaction during intermediate pages.
2.) The initial page loading is longer, but there are no intermediate page requests (better optimization).
3.) This is vastly different than anything anyone usually does, which means you need to be especially careful with documentation.
4.) This design concept facilitates totally stand-alone web software to be easily deployed without the web.
5.) You might be increasing complexity for avoiding page loads, but maybe not. I'm not sure.
All together, I think it just depends on what you want to accomplish. My coworker really just wanted to see if he could do it, which he could. However, how he did it was really pretty bad, to the point where everyone else connected his poor implementation with a poor idea it was fairly sad.
Mostly, I think if you follow good web design practices this wouldn't be too bad of a thing to pursue. What are your goals though?
I'm sorry I couldn't directly answer all of your questions. I hope my experiences are still helpful in answering if I think it is a bad idea or not.
-Brian J. Stinar-

SproutCore is one of the best out there when it comes to really being built from the ground up for single-page, potentially complex applications. Unlike some others like GWT or Cappucino, SproutCore is really about using JavaScript directly. They aren't the only one though. You might also want to look at JavaScriptMVC and qooxdoo.
Personally, I've built an extremely large and complex single-page application using JavaScript. It is currently around 100,000 lines (including comments/whitespace). To give a sense of scale, jQuery is around 6,000. To reach that size, I built my own framework, build tools etc. and it is extremely maintainable. You're asking if it can work, and I can tell you that it does, but you do need a little infrastructure if you're looking at doing something big. (BTW, there's a lot of lazy loading - its not 100,000 lines all at once!)
I also wouldn't really recommend this method for anything other than a web application. As pointed out, its still difficult for SEO, and may be strange for some users.

You're looking for ExtJS

I did my site like that in jquery, backed by spring mvc 3. For me it works fine and you have a lightweight, flashy feling while on it.
If you have time to dig in javascript seriously... it is really good exercise.
The only thing I really need to add is SEO, since spiders see only intial html. If it is important for you, i think backend static files and sitemap would be good enough solution.

I found something very light-weight that provides the basic needed functionally without forcing you into a whole framework. It's called Sammy and is inspired by Ruby's Sinatra: http://code.quirkey.com/sammy/
Very recommended!

Alternative: code the same in Java. Take a look to ItsNat
Think in JavaScript but code the same in Java in server with the same DOM APIs, in server is way easier to manage your application without custom client/bridges because UI and data are together.
Regarding SEO, bookmarking etc in single page there are solutions, take a look to the Single Page Interface Manifesto

Introductory JavaScript programming task for an expert developer

What would be a good mini-project to get intimate with JavaScript, as an advanced 'introduction' to the language? I want to actually code an application in JS, not hook up bits of it to enhance a web application.

A lot of stuff you could learn by doing an RSS reader on a page. Google shows what can be done. The whole lection concentrates on javascript, network access, security restrictions and medium data mangeling.
If you have the ability to do any sort of backend programming than AJAX is really neat to do. You can get a lot of good effects with less efforts. A good thing to build on up.

I would argue that if you're really an advanced programmer then the exercises above would not really give you any insight into the language as they are just variations on things you probably have already done. Javascript's strongest suit is it's LISP style ability to grow. Write something AI(ish) that creates new functions. Most people don't utilize the language in this way, but, its ability to augment its own classes on the fly is, I would argue, it's most unusual and most powerful feature.

Although not a project, watch the Douglas Crockford videos at YUI theater.
The biggest web based Javascript projects are going to deal with the DOM. Do some nifty stuff with JQuery. Make a table with rows that highlight when you hover. Make them update themselves through AJAX and JSON when you click on them.
If you're really looking for something magical and usefull write a scrollable table with fixed headers and footers for IE8.
If you want to stay away from the WEB use the JDK 1.6 and run Javascript code in that. You could do TONS with that.

Whenever I'm trying to get familiar with a language, I will work on Project Euler problems with it.

I would implement a simple game like sokoban first.
The second application would be an AJAX-based multiuser chat application, first fetching other people's responses by polling, later with AJAX push.

Interesting question.
Really you could do any sort of application. In order to make sure you're using the latest and greatest stuff, I'd try making a simple CRUD style application using DHTML and AJAX. Perhaps a contacts list or calendar. If you're feeling really energetic, you could write the back-end in JavaScript as well.
Unless you want to get really friendly with the DOM and browser compatibility, I'd learn Javascript through the mask of one of the nice frameworks like Jquery or Prototype.

The Holy Grail - a WYSIWYG editor. They wouldn't need to complete it, but just seeing their plan of attack would be interesting. Plays right into patterns and OO.

I suggest you create a Google Gadget. You can create one for free and perhaps make something useful out of it. If you don't have a Google account, sign up for one. Then add the Google Gadget Editor to begin writing your code.
With the gadget, you'll be able to mess with JavaScript, JSON, CSS, etc. Furthermore, you'll be able to store the file on Google's server so you can work on it from any computer.
I created a simple RSS reader and wrote JavaScript to get the feed (using Google's API) then dealing with that JavaScript object because it came back as JSON. I then developed some JavaScript to hide/show div tags.
It was a good starter project for me to learn JavaScript.

Get JavaScript the Good Parts by Douglas Crockford. Also check out his web site: http://www.crockford.com
Key reason: just because JavaScript looks like C/C++/Java/C# doesn't mean it actually is like them. Things are significantly different. I suggest reading his book to get a grasp of those differences.
Otherwise, I would look at the JQuery web site. JavaScript is cool and all, but a good framework will save you from a lot of the pitfalls and make you much more productive faster.

try making an advanced AJAX application like for example try to recreate the google calander.

How about a firefox plugin to monitor StackOverflow? It could use RSS to monitor feeds and let you know when new questions are asked with your tags.
It could also be grown as your js skills progress.

Write yet another javascript framework, but focused specially in something, ie game programming.

Should I use Google Web Toolkit for my new webapp?

I would like to create a database backed interactive AJAX webapp which has a custom (specific kind of events, editing) calendaring system. This would involve quite a lot of JavaScript and AJAX, and I thought about Google Web Toolkit for the interface and Ruby on Rails for server side.
Is Google Web Toolkit reliable and good? What hidden risks might be if I choose Google Web Toolkit? Can one easily combine it with Ruby on Rails on server side? Or should I try to use directly a JavaScript library like jQuery?
I have no experience in web development except some HTML, but I am an experienced programmer (c++, java, c#), and I would like to use only free tools for this project.

RoR is actually one of the things the GWT is made to work well with, as long as you're using REST properly. It's in the Google Web Toolkit Applications book, and you can see a demo from the book using this kind of idea here. That's not to say that you won't have any problems, but I think the support is definitely out there for it.
There's a neat project for making RoR/GWT easy that you can find here (MIT license). I haven't had a chance to try it out yet, but it looks like a good amount of thought has been put into it. One catch is that it looks like it hasn't been fully tested with 2.1 Rails yet, just 2.0, so you may run into a few (probably minor and fixable) errors.

If you are looking to integrate GWT with non-Java backends such as ROR, PHP etc., you should bear in mind that GWT 1.5 now supports JavaScript Overlay types. This feature lets you write classes that can be mapped over the top of native JavaScript objects to easily provide accessor methods for properties of those objects and other extended functionality.
See this link for more details:
JavaScript Overlay Types
So you could return JSON encoded data from your backend via AJAX calls, parse it into a JavaScript Object and then access the data through your GWT Java code using the overlay classes you've created. Or when you render your page you can render static config data as JavaScript Objects and read it in via this mechanism, rather than having to do an AJAX call to grab the data.

If you know JAVA, and have somewhere you can host it (like a tomcat or glassfish container) I would recommend that much more than using Ruby for the back end. The main reason is that then you can share all of your objects, and use the built in RPC mechanism. I've done this for quite a lot of our projects and it's a huge timesaver, not to mention that the code is less error prone, because you don't convert your java objects to anything and then back again.
I have linked my GWT with Rails before, using the to_json function in Rails and then reading the JSON in GWT. It's all supported, but it is far more annoying than just doing the back end in JAVA.
Of course if you have cheap hosting, then Java containers are pretty much out of the question, in which case I would think Rails would be the next best thing.

GWT is very high quality with a great community. However you do need to know CSS if you want to adjust the look of things (you will) - CSS can do a lot of the layout, just like regular web if you want it to. Libraries like GWT-ext or ExtGWT can help a bit as they have stunning "out of the box" looks but for a price (extra size to your app).

You can code everything in Java using GWT, and you can integrate existing 3rd party javascript libraries with it. It's very good. I've never used RoR much though, so can't say anything about that.

If you're experienced in Java but not in Javascript/CSS, then GWT is going to be a lifesaver (unless you want to learn them, of course). CSS has so many little fiddly details. It is not uncommon to spend half a day fixing a 2 pixel misalignment that only occurs in IE6.
I am not sure about how easy it would be to use ROR for the back end... It is possible, I am sure, since GWT ajax communication is just servlets. But they provide some really nice functionality for passing Java objects back and forth which you won't be able to utilize if your server isn't also using Java.

I wrote about some of the disadvantages of GWT recently. Mainly, the disadvantages are: long deployment cycle for changes to some parts of the application and a rather steep learning curve. As a seasoned Java programmer, the second should be less of a problem and if you use a seperate backend, the first is also mitigated (as a complete redeploy is primarily required when you change the 'server' part of the application).

GWT is a wonderful framework with lots of potential. Keep in mind that it's still quite new, though. There are some unresolved bugs that can really annoy you, and they usually require ugly workarounds to get past. The community is great but you'll probably end up with a few problems sooner or later that Google can't answer yet.
But hey, I say go for it. The potential for GWT is awesome, and I bet it's future will be bright.

You should definitely use GWT for a new project (it's pretty easy to use in an old project too).
I my experience it's very fast to learn and use. The compiled javascript code is much better than anything you could ever write by hand and it works fast too.
Another benefit is the ability to debug you're code (which is hell with javascript alone)

This blog has inputs from many experienced users of GWT and have some great discussion points. I personally have huge experience with varied UI Frameworks. I will add my two cents. Lets look at fundamental advantages and disadvantages of GWT
Fundamental Advantage
GWT takes the web layer programming to JAVA. So, the obvious advantages of Java start getting into play. It will provide Object Oriented programming. It will also provide great debugging and compile time checks. Since it generates HTML and Javascript, it will also have ability to hide some complexity within its generator.
Fundamental Disadvantage
The disadvantage starts from the same statement. GWT takes the web layer programming to JAVA. If you know JAVA, probably you will never look out for an alternative language to write your business logic. It's self sufficient and great. But when it comes to writing configurations for a JAVA application. We use property files, database, XML etc. We never store configurations in a JAVA class file. Think hard, why is that?
This is because configuration is a static data. It often require hierarchy. It is supposed to be readable. It never requires compilation. It doesn't require knowledge of JAVA programming language. In short, it is a different ball game. Now the question is, how it relates to our discussion?
Now, lets think about a web page. Do you think when we write a web page we write a business logic? Absolutely not. Web page is just a configuration. It is a configuration of hierarchical containers and fields. We need to write business logic for the data that will be captured from and displayed on the web page and not to create the web page itself.
Previous paragraph makes a very very strong statement. This will explain why HTML and XML based web pages are still the most popular ones. XML is the best in business to write configurations. A framework must allow a clear separation of web page from business logic (the goal of MVC framework). By doing this a web designer will be able to apply his skills of visualization and artistry to create brilliant looking web pages just by configuring XMLs and without being bothered about the intricacies of a programming language. Developers will be able to use their best in business JAVA for writing business logic.
Finally, lets talk about the repercussions in direct terms. GWT breaks this principal so it is bound to fail. The cost for developing GWT application will be very high because you will need multiskill programmers to write web pages. The required look and feel will be very hard to achieve. The turn around time of modifying the web page will be very high because of unnecessary compilation. And lastly, since you are writing web pages in JAVA it is very easy to corrupt it with business logic. Unknowingly you will introduce complexities that must be avoided.

You could also consider Grails ("Groovy on Rails") which gives you the benefits of a Rails framework and the use of the Java VM.

Our team recently asked the same question, and we chose to go with GWT, especially since the designer plugin made working with GWT more accessible to non-java experts on the team. Whoever makes this choice, just beware you DON'T use the GWT Designer plugin !! It has not been updated (in at least a year, apparently) to create a GWT application that is compatible with IE8.
Our team had almost completed our application layouts, which were working perfectly in Chrome, FF and Safari. Then they blew up in IE. IE 7 would load partial pages (but not composite includes), and IE8 was not even able to load up the application. It just hung.
The designer plugin has buttons that allow the user to add CellTable widgets that are not IE compatible (CellTable, DeckPanel, Horizontal Panel, Vertical Panel, among others). These will cause intense pain when the layouts have to be re-done in java without assistance from the designer.
Experienced GWT users love it, but the designer plugin will kill you.