We Keep Coding

Load a DOM and Execute javascript, server side, with .Net

I would like to load a DOM using a document (in string form) or a URL, and then Execute javascript functions (including jquery selectors) against it. This would be totally server side, in process, no client/browser.
Basically I need to load the dom and then use jquery selectors and text() & type val() functions to extract strings from it. I don't really need to manipulate the dom.
I have looked at .Net javascript engines such as Jurassic and Jint, but neither support loading a DOM, and so therefore can't do what I need.
I would be willing to consider non .Net solutions (node.js, ruby, etc) if they exist, but would really prefer .Net.
edit
The below is a good answer, but currently I'm trying a different route, I'm attempting to port envjs to jurassic. If I can get that working I think it will do what I want, stay tuned....

The answer depends on what you are trying to do. If your goal is basically a complete web browser simulation, or a "headless browser," there are a number of solutions, but none of them (that I know of) exist cleanly in .NET. To mimic a browser, you need a javascript engine and a DOM. You've identified a few engines; I've found Jurassic to be both the most robust and fastest. The google chrome V8 engine is also very popular; the Neosis Javascript.NET project provides a .NET wrapper for it. It's not quite pure .NET since you have a non-.NET dependency, but it integrates cleanly and is not much trouble to use.
But as you've noted, you still need a DOM. In pure C# there is XBrowser, but it looks a bit stale. There are javascript-based representations of the entire browser DOM like jsdom, too. You could probably run jsdom in Jurassic, giving you a DOM simulation without a browser, all in C# (though likely very slowly!) It would definitely run just fine in V8. If you get outside the .NET realm, there are other better-supported solutions. This question discusses HtmlUnit. Then there's Selenium for automating actual web browsers.
Also, bear in mind that a lot of the work done around the these tools is for testing. While that doesn't mean you couldn't use them for something else, they may not perform or integrate well for any kind of stable use in inline production code. If you are trying to basically do real-time HTML manipulation, then a solution mixing a lot of technologies not that aren't widely used except for testing might be a poor choice.
If your need is actually HTML manipulation, and it doesn't really need to use Javascript but you are thinking more about the wealth of such tools available in JS, then I would look at C# tools designed for this purpose. For example HTML Agility Pack, or my own project CsQuery, which is a C# jQuery port.
If you are basically trying to take some code that was written for the client, but run it on a server -- e.g. for sophisticated/accelerated web scraping -- I'd search around using those terms. For example this question discusses this, with answers including PhantomJS, a headless webkit browser stack, as well as some of the testing tools I have already mentioned. For web scraping, I would imagine you can live without it all being in .NET, and that may be the only reasonable answer anyway.

Totally JavaScript Web Stack - Middleware, Webserver, DB suggestions?

I'm currently looking for a new web stack to build a hobby project on and would like it to be powered by JavaScript. I've had a quick look at Nitro, NarwhalJS etc. but was wondering if anyone had any solid recommendations or experience of an entire end-to-end javascript/json architecture ( jquery, middleware, standard libraries, db etc.) that they could share.
I'd prefer it to be a stack that you think is going to grow and is actively being looked after, documentation, community of nice like-minded individuals etc.
Thanks in advance.

Caveat: This answer somewhat fails to meet your basic requirement of personal experience with the resources listed. :-)
Off-the-cuff, there's Jaxer from Aptana and Chrome Server (which, despite the name, isn't related to Google Chrome AFAIK). Speaking of Chrome, though, there's an Apache CGI module that supports server-side scripting using JavaScript via Google's V8 engine, but that doesn't help you much with DB connectivity and such.
I'll also plug Java and Rhino, which I have used a bit. Via Rhino, you can compile JavaScript into Java bytecode (which, if you're using the Sun runtime, is JIT-compiled to machine code by Hotspot). That means you can run JavaScript in any servlet container (Tomcat, Resin, etc.). The joy here is that all of the huge array of goodies available for Java is instantly available to you via JavaScript -- so, MySQL connectors, image manipulation libraries, just about anything you can imagine. It's also amazingly easy to access those resources from JavaScript via Rhino. To give you an idea:
importPackage(java.io);
var f = new FileWriter("test.txt");
f.write("Testing 1 2 3");
f.close();
...and we've just written a file on the server via JavaScript, using Java's java.io.FileWriter class. You can also execute dynamic scripts at runtime via the javax.script package, which (for JavaScript) uses Rhino under the covers, although I'm not immediately coming up with a use case for doing that. :-)

For the database part of the stack:
Couchdb uses JSON and REST to store data in a document format. It uses PUT,DELETE for storage - I'm not sure how that would work with Javascript.

Helma should work well as a web server layer. It streamlines the use of Rhino as the web tier logic language.

I'm building a new service called PageForest that helps developers write totally client-side javascript programs, with PageForest providing storage and user management. Here's a sample page:
http://pfsamples.googlecode.com/svn/trunk/SAMTable/index.html#mckoss_16
This is still a work in progress, but I'd love to get some feedback on the approach. You can find more samples at the pfsamples.googlecode.com site.

Check out JScript / Windows Scripting Host(wsh) and possibly HTA's. HTA's can actually be served via a webserver and act as a locally running application with extended rights. If you want pure web development you can use WSH and some IIS tweaking to process server requests on the back end with pure javascript code in a WSF file. WSH also provides access to the file system, ODBC compliant databases and a slew of other COM exposed applications via the ActiveX model. We're not talking blazing speed, but you're programming in javascript to begin with.
Here are some links on the "stack"
http://msdn.microsoft.com/en-us/library/ms536496(VS.85).aspx
http://msdn.microsoft.com/en-us/library/15x4407c(VS.85).aspx

How about using
GWT-Spring-Hibernate-MYSql

Server Side Javascript: Why?

Is the use of server side javascript prevalent? Why would one use it as opposed the any other server side scripting? Is there a specific use case(s) that makes it better than other server side languages?
Also, confused on how to get started experimenting with it, I'm on freeBSD, what would I need installed in order to run server side javascript?

It goes like this:
Servers are expensive, but users will give you processing time in their browsers for free. Therefore, server-side code is relatively expensive compared to client-side code on any site big enough to need to run more than one server. However, there are some things you can't leave to the client, like data validation and retrieval. You'd like to do them on the client, because it means faster response times for the users and less server infrastructure for yourself, but security and accessibility concerns mean server-side code is required.
What typically happens is you do both. You write server-side logic because you have to, but you also write the same logic in javascript in hopes of providing faster responses to the user and saving your servers a little extra work in some situations. This is especially effective for validation code; a failed validation check in a browser can save an entire http request/response pair on the server.
Since we're all (mostly) programmers here we should immediately spot the new problem. There's not only the extra work involved in developing two sets of the same logic, but also the work involved in maintaining it, the inevitable bugs resulting from platforms don't match up well, and the bugs introduced as the implementations drift apart over time.
Enter server-side javascript. The idea is you can write code once, so the same code runs on both server and client. This would appear to solve most of the issue: you get the full set of both server and client logic done all at once, there's no drifting, and no double maintenance. It's also nice when your developers only need to know one language for both server and client work.
Unfortunately, in the real world it doesn't work out so well. The problem is four-fold:
The server view of a page is still very different from the client view of a page. The server needs to be able to do things like talk directly to a database that just shouldn't be done from the browser. The browser needs to do things like manipulate a DOM that doesn't match up with the server.
You don't control the javascript engine of the client, meaning there will still be important language differences between your server code and your client code.
The database is normally a bigger bottleneck than the web server, so savings and performance benefit ends up less than expected.
While just about everyone knows a little javascript, not many developers really know and understand javascript well.
These aren't completely unassailable technical problems: you constrain the server-supported language to a sub-set of javascript that's well supported across most browsers, provide an IDE that knows this subset and the server-side extensions, make some rules about page structure to minimize DOM issues, and provide some boiler-plate javascript for inclusion on the client to make the platform a little nicer to use. The result is something like Aptana Studio/Jaxer, or more recently Node.js, which can be pretty nice.
But not perfect. In my opinion, there are just too many pitfalls and little compatibility issues to make this really shine. Ultimately, additional servers are still cheap compared to developer time, and most programmers are able to be much more productive using something other than javascript.
What I'd really like to see is partial server-side javascript. When a page is requested or a form submitted the server platform does request validation in javascript, perhaps as a plugin to the web server that's completely independent from the rest of it, but the response is built using the platform of your choice.

I think a really cool use of server-side Javascript that isn't used nearly often enough is for data validation. With it, you can write one javascript file to validate a form, check it on the client side, then check it again on the server side because we shouldn't trust anything on the client side. It lets you keep your validation rules DRY. Quite handy.
Also see:
Will server-side Javascript take off? Which implementation is most stable?
When and how do you use server side JavaScript?

Javascript is just a language. Because it is just a language, you can use it anywhere you want... in your browser, on the server, embedded in other applications, stand-alone applications, etc.
That being said, I don't know that there is a lot of new development happening with "Server-Side Javascript"

Javascript is a perfectly good language with a self / scheme prototype style base and a C style syntax. There are some problems, see Javascript the Good Parts, but in general it's a first rate language. The problem is that most javascript programmers are terrible programmers because it's very accessible to get started.
One team at google built out Rhino on Rails, which is an MVC framework like Ruby on Rails which is written in javascript and runs on Rhino a javascript interpreter for the Java VM. In this case they had a requirement to use the Java VM, but wanted to get a language which was fast (javascript is fast), supported duck typing, and was flexible.
Another example is something like CouchDB, a document oriented database which uses json as it's transport format and javascript as it's query & index language. They wanted the database to be as web native as possible.
Javascript is good at string and dom (xml) manipulation, being sandboxed, networking, extending itself, etc... Those kind of features are the thing you often do when developing web applications.
All that said, i don't actually develop server side javascript. It's not a bad idea, but definitely less common.

We use javascript on the client because it is there, not because from a list of languages it was our choice. I wouldn't choose it for any chore on the server.
You can run any language you like on the server, in fact, as many as you like.
javascript is reliable and easy to use, but it is just too labor intensive for common tasks on the server.

I have used both Javascript (NodeJS) and compiled languages (such as Java or C#.NET). There are huge discussions on the internet about which is preferable. This question is very old (2009). Since then the Javascript world has changed considerably, whereas honestly the Java world has not changed much (relatively).
There have been huge advancements in the Javascript world with Typescript, amazing frameworks (such as Next.JS), reactive programming, functional programming, GraphQL, SSR etc.
When I look at compiled code, especially Java code it all still seems to be the same old tools - Spring (maybe SpringBoot) and Jackson. .NET has advanced server side, but not to the extent of the JS world.
Sure my list there can be used with several languages, but I believe Javascript has improved the software engineering world considerably.
Server side development with Javascript, Typescript and NodeJs is engaging, fun and productive. Use it and enjoy it. Like millions are today.

GWT without JavaScript?

I was looking into GWT. It seems nice, but our software have the must work without JS requirement. Is it possible?

No, it isn't. GWT provides a windowing toolkit that is specifically designed to run on the client, not on the server. Degraded (e.g. non-javascript) code would need to deliver complete HTML to the browser, which GWT simply does not do. It compiles your java code to a javascript file that is delivered to the client and builds the UI by DOM-manipulation on the client. Then there's some code to talk back to the server, some implicit, some written by you yourself. This model does not lend itself well to degrading gracefully.
The only way to degrade somewhat gracefully is to provide a second, non-javascript UI or use another toolkit that doesn't render the frontend on the client but delivers HTML. Sorry.

You could degrade gracefully by creating an html structure that is just 'good enough' (with form posts, linked menus, etc) and then have GWT attach to each part of that structure, augmenting its behavior. For example, make an HTML drop down dynamic, replace a link to another page with a component that opens a lightbox, or replace a link to another page with an XML http request to do the same thing (e.g. cast a vote).
I've done this a number of times for clients.
It's the opposite way that most GWT gets developed, but it can work.

I was looking at this issue myself when designing my website. GWT isn't really any better than just writing Javascript files in that their syntax is almost identical. The true benefit comes when you share client and server libraries. Hopefully you've resolved this issue in the last two years, but at any rate here are a couple examples that you may find useful.
Creating Gmail: With GWT, you can create an EmailFormatter in a shared package that does the email listing markup so that your server doesn't have to. You could then add support for legacy browsers ("older version") by using the same EmailFormatter class on the server side.
Form verification: While is is absolutely necessary from a security perspective to validate user input server side, it is more convenient for most users to have Javascript check a form before it is submitted. You can use the same Java code with GWT to do this.

Should I use Google Web Toolkit for my new webapp?

I would like to create a database backed interactive AJAX webapp which has a custom (specific kind of events, editing) calendaring system. This would involve quite a lot of JavaScript and AJAX, and I thought about Google Web Toolkit for the interface and Ruby on Rails for server side.
Is Google Web Toolkit reliable and good? What hidden risks might be if I choose Google Web Toolkit? Can one easily combine it with Ruby on Rails on server side? Or should I try to use directly a JavaScript library like jQuery?
I have no experience in web development except some HTML, but I am an experienced programmer (c++, java, c#), and I would like to use only free tools for this project.

RoR is actually one of the things the GWT is made to work well with, as long as you're using REST properly. It's in the Google Web Toolkit Applications book, and you can see a demo from the book using this kind of idea here. That's not to say that you won't have any problems, but I think the support is definitely out there for it.
There's a neat project for making RoR/GWT easy that you can find here (MIT license). I haven't had a chance to try it out yet, but it looks like a good amount of thought has been put into it. One catch is that it looks like it hasn't been fully tested with 2.1 Rails yet, just 2.0, so you may run into a few (probably minor and fixable) errors.

If you are looking to integrate GWT with non-Java backends such as ROR, PHP etc., you should bear in mind that GWT 1.5 now supports JavaScript Overlay types. This feature lets you write classes that can be mapped over the top of native JavaScript objects to easily provide accessor methods for properties of those objects and other extended functionality.
See this link for more details:
JavaScript Overlay Types
So you could return JSON encoded data from your backend via AJAX calls, parse it into a JavaScript Object and then access the data through your GWT Java code using the overlay classes you've created. Or when you render your page you can render static config data as JavaScript Objects and read it in via this mechanism, rather than having to do an AJAX call to grab the data.

If you know JAVA, and have somewhere you can host it (like a tomcat or glassfish container) I would recommend that much more than using Ruby for the back end. The main reason is that then you can share all of your objects, and use the built in RPC mechanism. I've done this for quite a lot of our projects and it's a huge timesaver, not to mention that the code is less error prone, because you don't convert your java objects to anything and then back again.
I have linked my GWT with Rails before, using the to_json function in Rails and then reading the JSON in GWT. It's all supported, but it is far more annoying than just doing the back end in JAVA.
Of course if you have cheap hosting, then Java containers are pretty much out of the question, in which case I would think Rails would be the next best thing.

GWT is very high quality with a great community. However you do need to know CSS if you want to adjust the look of things (you will) - CSS can do a lot of the layout, just like regular web if you want it to. Libraries like GWT-ext or ExtGWT can help a bit as they have stunning "out of the box" looks but for a price (extra size to your app).

You can code everything in Java using GWT, and you can integrate existing 3rd party javascript libraries with it. It's very good. I've never used RoR much though, so can't say anything about that.

If you're experienced in Java but not in Javascript/CSS, then GWT is going to be a lifesaver (unless you want to learn them, of course). CSS has so many little fiddly details. It is not uncommon to spend half a day fixing a 2 pixel misalignment that only occurs in IE6.
I am not sure about how easy it would be to use ROR for the back end... It is possible, I am sure, since GWT ajax communication is just servlets. But they provide some really nice functionality for passing Java objects back and forth which you won't be able to utilize if your server isn't also using Java.

I wrote about some of the disadvantages of GWT recently. Mainly, the disadvantages are: long deployment cycle for changes to some parts of the application and a rather steep learning curve. As a seasoned Java programmer, the second should be less of a problem and if you use a seperate backend, the first is also mitigated (as a complete redeploy is primarily required when you change the 'server' part of the application).

GWT is a wonderful framework with lots of potential. Keep in mind that it's still quite new, though. There are some unresolved bugs that can really annoy you, and they usually require ugly workarounds to get past. The community is great but you'll probably end up with a few problems sooner or later that Google can't answer yet.
But hey, I say go for it. The potential for GWT is awesome, and I bet it's future will be bright.

You should definitely use GWT for a new project (it's pretty easy to use in an old project too).
I my experience it's very fast to learn and use. The compiled javascript code is much better than anything you could ever write by hand and it works fast too.
Another benefit is the ability to debug you're code (which is hell with javascript alone)

This blog has inputs from many experienced users of GWT and have some great discussion points. I personally have huge experience with varied UI Frameworks. I will add my two cents. Lets look at fundamental advantages and disadvantages of GWT
Fundamental Advantage
GWT takes the web layer programming to JAVA. So, the obvious advantages of Java start getting into play. It will provide Object Oriented programming. It will also provide great debugging and compile time checks. Since it generates HTML and Javascript, it will also have ability to hide some complexity within its generator.
Fundamental Disadvantage
The disadvantage starts from the same statement. GWT takes the web layer programming to JAVA. If you know JAVA, probably you will never look out for an alternative language to write your business logic. It's self sufficient and great. But when it comes to writing configurations for a JAVA application. We use property files, database, XML etc. We never store configurations in a JAVA class file. Think hard, why is that?
This is because configuration is a static data. It often require hierarchy. It is supposed to be readable. It never requires compilation. It doesn't require knowledge of JAVA programming language. In short, it is a different ball game. Now the question is, how it relates to our discussion?
Now, lets think about a web page. Do you think when we write a web page we write a business logic? Absolutely not. Web page is just a configuration. It is a configuration of hierarchical containers and fields. We need to write business logic for the data that will be captured from and displayed on the web page and not to create the web page itself.
Previous paragraph makes a very very strong statement. This will explain why HTML and XML based web pages are still the most popular ones. XML is the best in business to write configurations. A framework must allow a clear separation of web page from business logic (the goal of MVC framework). By doing this a web designer will be able to apply his skills of visualization and artistry to create brilliant looking web pages just by configuring XMLs and without being bothered about the intricacies of a programming language. Developers will be able to use their best in business JAVA for writing business logic.
Finally, lets talk about the repercussions in direct terms. GWT breaks this principal so it is bound to fail. The cost for developing GWT application will be very high because you will need multiskill programmers to write web pages. The required look and feel will be very hard to achieve. The turn around time of modifying the web page will be very high because of unnecessary compilation. And lastly, since you are writing web pages in JAVA it is very easy to corrupt it with business logic. Unknowingly you will introduce complexities that must be avoided.

You could also consider Grails ("Groovy on Rails") which gives you the benefits of a Rails framework and the use of the Java VM.

Our team recently asked the same question, and we chose to go with GWT, especially since the designer plugin made working with GWT more accessible to non-java experts on the team. Whoever makes this choice, just beware you DON'T use the GWT Designer plugin !! It has not been updated (in at least a year, apparently) to create a GWT application that is compatible with IE8.
Our team had almost completed our application layouts, which were working perfectly in Chrome, FF and Safari. Then they blew up in IE. IE 7 would load partial pages (but not composite includes), and IE8 was not even able to load up the application. It just hung.
The designer plugin has buttons that allow the user to add CellTable widgets that are not IE compatible (CellTable, DeckPanel, Horizontal Panel, Vertical Panel, among others). These will cause intense pain when the layouts have to be re-done in java without assistance from the designer.
Experienced GWT users love it, but the designer plugin will kill you.