We Keep Coding

Load a DOM and Execute javascript, server side, with .Net

I would like to load a DOM using a document (in string form) or a URL, and then Execute javascript functions (including jquery selectors) against it. This would be totally server side, in process, no client/browser.
Basically I need to load the dom and then use jquery selectors and text() & type val() functions to extract strings from it. I don't really need to manipulate the dom.
I have looked at .Net javascript engines such as Jurassic and Jint, but neither support loading a DOM, and so therefore can't do what I need.
I would be willing to consider non .Net solutions (node.js, ruby, etc) if they exist, but would really prefer .Net.
edit
The below is a good answer, but currently I'm trying a different route, I'm attempting to port envjs to jurassic. If I can get that working I think it will do what I want, stay tuned....

The answer depends on what you are trying to do. If your goal is basically a complete web browser simulation, or a "headless browser," there are a number of solutions, but none of them (that I know of) exist cleanly in .NET. To mimic a browser, you need a javascript engine and a DOM. You've identified a few engines; I've found Jurassic to be both the most robust and fastest. The google chrome V8 engine is also very popular; the Neosis Javascript.NET project provides a .NET wrapper for it. It's not quite pure .NET since you have a non-.NET dependency, but it integrates cleanly and is not much trouble to use.
But as you've noted, you still need a DOM. In pure C# there is XBrowser, but it looks a bit stale. There are javascript-based representations of the entire browser DOM like jsdom, too. You could probably run jsdom in Jurassic, giving you a DOM simulation without a browser, all in C# (though likely very slowly!) It would definitely run just fine in V8. If you get outside the .NET realm, there are other better-supported solutions. This question discusses HtmlUnit. Then there's Selenium for automating actual web browsers.
Also, bear in mind that a lot of the work done around the these tools is for testing. While that doesn't mean you couldn't use them for something else, they may not perform or integrate well for any kind of stable use in inline production code. If you are trying to basically do real-time HTML manipulation, then a solution mixing a lot of technologies not that aren't widely used except for testing might be a poor choice.
If your need is actually HTML manipulation, and it doesn't really need to use Javascript but you are thinking more about the wealth of such tools available in JS, then I would look at C# tools designed for this purpose. For example HTML Agility Pack, or my own project CsQuery, which is a C# jQuery port.
If you are basically trying to take some code that was written for the client, but run it on a server -- e.g. for sophisticated/accelerated web scraping -- I'd search around using those terms. For example this question discusses this, with answers including PhantomJS, a headless webkit browser stack, as well as some of the testing tools I have already mentioned. For web scraping, I would imagine you can live without it all being in .NET, and that may be the only reasonable answer anyway.

Avoiding writing the same algorithm in multiple different languages

I am a web developer, and I have observed that many times I need the same function on both client and server. So I write it in JS as well as in PHP or whichever server side language. I am fed up with this. If I have to change it then I need to change it in both places. If I want to use it for some hand held devices, then I will have to rewrite that code yet again using objective-C or Java etc. Then if I need to change that function then I will need to change it everywhere.
Is there a solution for this? If I will call some webservice via ajax, then the client will have a delay. If it will be in JS then it can't be accessed from within PHP or Java, etc. If I use some service in PHP from another language then that can also become a performance issue.
It is also possible that some time we need such functions output from some paramters as input using db or without db.
I know there would be some pretty simpler solution but I am not aware of that. Please tell some language independent solution as I don't have VPS always.
I am not sure if my question actually belongs to stackoverflow.com or programmers.stackexchange.com so please transfer it to programmers.stackexchange.com instead of closing this question if it belongs to there.

Typically, the solution to this problem is to write common code in one language and use translators or library linking to allow access from other languages.
Node.js allows you to write server-side code in JavaScript.
Node.js is a platform built on Chrome's JavaScript runtime for easily building fast, scalable network applications. Node.js uses an event-driven, non-blocking I/O model that makes it lightweight and efficient, perfect for data-intensive real-time applications that run across distributed devices.
You can also use JavaScript to write HTML5 apps for mobile devices.
"Building iPhone Apps with HTML, CSS, and JavaScript"
Now web designers and developers can join the iPhone app party without having to learn Cocoa's Objective-C programming language. It's true: You can write iPhone apps quickly and efficiently using your existing skills with HTML, CSS, and JavaScript. This book shows you how with lots of detailed examples, step-by-step instructions, and hands-on exercises.
If you don't want to try to write large complex applications in JavaScript, GWT provides a way to write Java and via-translation, run it on the client.
The GWT SDK contains the Java API libraries, compiler, and development server. It lets you write client-side applications in Java and deploy them as JavaScript.
If you develop in .Net languages: C# -> JavaScript ScriptSharp
Script# is a free tool that enables developers to author C# source code and subsequently compile it into regular script that works across all modern browsers

you could use the spidermonkey extension to translate php into javascript. this way you can write your functions in php then simply convert them to javascript and re-use them at the browser.
here is a good tutorial to show you how this is done

Execute javascript on IIS server

I have the following situation. A customer uses JavaScript with jQuery to create a complex website. We would like to use JavaScript and jQuery on the server (IIS) for the following reasons:
Skills transfer - we would like to use JavaScript and jQuery on the server and not have to use eg VB Script. / classic asp. .Net framework/Java etc is ruled out because of this.
Improved options for search/accessibility. We would like to be able to use jQuery as a templating system, but this isn't viable for search engines and users with js turned off - unless we can selectively run this code on the server.
There is significant investment in IIS and Windows Server, so changing that is not an option.
I know you can run jScript on IIS using windows Script host, but am unsure of the scalability and the process surrounding this. I am also unsure whether this would have access to the DOM.
Here is a diagram that hopefully explains the situation. I was wondering if anyone has done anything similar?
EDIT: I am not looking for critic on web architecture, I am simply wanting to know if there are any options for manipulating the DOM of a page before it is sent to the client, using javascript. Jaxer is one such product (no IIS) Thanks.

Have a look at bringing the browser to the server, Rhino, and Use Microsoft's IIS as a Java servlet engine.
The first link is from John Resig's (jQuery's creator) blog.
Update August 2 2011
Node.js is coming to Windows.

The idea to reuse client JS on the server may sound tempting, but I am not sure that jQuery itself would be ready to run in server environment.
You will need to define global context for jQuery somehow by initializing window, document, self, location, etc.. I am not sure it is doable.
Besides, as Cheeso has mentioned, Active Server Pages is a very outdated technology, it was replaced with ASP.Net by Microsoft in the beginning of the century. I used to maintain a legacy system using ASP 3.0 for more than a year and that was pain. The most wonderful pastime was debugging: you will hardly find anything for the purpose today and will have to decript beautiful errors like in IIS log:
error '800a9c68'
Application-defined or object-defined error
Nevertheless, I can confirm that I managed to reuse client and server JScript. But this was code written by me who knew that it was going to be used on the server.
P.S. I would not recommend move that way. There are plenty templating frameworks which are familiar to those who write HTML and JavaScript.

JScript runs on IIS via something called ASP.
Active Server Pages.
It was first available in 1996.
Eventually ASP.NET was introduced as a successor. But ASP is still supported.
There is no DOM for the HTML page, though.
You might need to reconsider your architecture a bit.

I think the only viable solutions you're likely to find anywhere near ready to go involve putting IIS in front of Java. There are two browser-like environments I'm aware of coded for Java:
1) Env-js (see http://groups.google.com/group/envjs and http://github.com/thatcher/env-js )
I believe this one has contributions from jQuery's John Resig and was put together with jQuery testing/support in mind.
2) HTMLUnit (see http://htmlunit.sourceforge.net/ ) This one's older, and wasn't originally conceived around jQuery, but there are reports in the wild of using it to run jQuery's test suite successfully (http://daniel.gredler.net/2007/08/08/htmlunit-taming-jquery/ ).
If you want something pure-IIS/MS, I think your observation about windowsScript host and/or something like the semi-abandoned JScript.NET is probably about as close as you're going to come, along with a port (which you'll probably have to start) of something like Env-js or HTMLUnit.
Also, I don't know if you've seen the Wikipedia list of server-side JavaScript solutions:
http://en.wikipedia.org/wiki/Server-side_JavaScript
Finally... you could probably write a serviceable jQuery-like library in any language that already has some kind of DOM library and first-class functions (or, failing that an eval facility). See, for example pQuery for Perl (http://metacpan.org/pod/pQuery ). This would get you the benefits of the jQuery style of manipulating documents. Skill transfer is great and JavaScript has a wonderful confluence of very nice features, but on the other hand, having developers who care enough to learn multiple languages is also great, and js isn't the only nice language out there.

I think it's mainly a browser based script so probably you are better of using technologies based on VB or .NET to perform or generate HTML from templates. I'm sure there are because in the java world there are a few of these around (like velocity). You'd then use jQuery to create or add client side functionality and usability so it makes the website more usable than it would have been.

What exactly do you mean by
"A customer uses JavaScript with
jQuery to create a complex website"
Half the point of jQuery is to make it easy for the developer to manipulate the DOM, and therefore add interactive enhancements to a web site. By running the Javascript on the server and only rendering HTML you will lose the ability to add these enhancements, without doing a round trip to the server (think WebForms postback model...ugh).
Now if what you really mean is the customer uses a site builder based on jQuery, why not have that tool output flat HTML in the first place?

Take a look at this technology. You can invoke scripts to run at server, at client, or both. Plus, this really implements the firefox engine on the server. Take a look at it.
Aptana's Jaxer is the first AJAX web server so far. I have not tryed it yet, but I will. Looks promising and very powerful.

How to Cross-Compile Java Source Code to JavaScript?

Given a set of Java source code files, how can I compile them into one or more JavaScript files that can be used with hand-crafted JavaScript?
GWT is one option, but every example I've seen so far is aimed at building fancy websites. The simple use case of just converting Java source to Javascript that can be used together with handcrafted JavaScript hasn't been well-documented.
I started a thread on the GWT mailing list on this subject, but opinions seem to be mixed on whether this is even feasible.
One person gave a very useful tip, which was to check out GWT-Exporter. The problem is that neither source code nor documentation is readily available, although there's this and this.
edit: GWT-Exporter source code is here
I've also seen Java2Script. But again, I wasn't able to find examples of how to solve my simple use case.
What's the best approach to this problem? Is there something better I'm missing?

When you use GWT, you're basically converting the UI portion into Javascript (and it assumes that you use the UI widgets provided when you write your Java). Only some of the Java libraries are accessible within Javascript. Typically in a GWT application anything that makes heavy use of Java libraries would run on the server side and connect to the Javascript as AJAX (which GWT handles for you). So GWT isn't necessarily converting your full application into Javascript (though it can if you're willing to limit your use of Java libraries and some functionality).
At any rate, if this approach (calling out to Java running on a server from within Javascript) appeals to you, one nice option is DWR, which basically allows your Javascript to directly call methods in Java classes running on the server (without you having to build a web service or other frontend). Not what you asked, I know.
More relevantly, it looks like there's source code for a sample app demonstrating the use of gwt-exporter.

I am not sure if it fits your use case, but if you agree to drop Java APIs and use JavaScript APIs from Java, then you can use JSweet, a Java to JavaScript transpiler built on the top of TypeScript. It gives you access to hundreds of well-typed JavaScript APIs (DOM, jQuery, underscore, angularjs, etc). It generates JavaScript code and you can mix it with legacy JavaScript and TypeScript code.
Note: JSweet will not work for legacy Java code and legacy Java APIs, but your use case did not mention reusing legacy code.
[UPDATE] Since version 1.1, JSweet now also supports some Java APIs such as Collections (java.util). So, it is possible to reuse legacy Java code to a certain extent. It is also quite straightforward to add your own support for Java APIs.

While the question is about compiling Java sources to JavaScript I think it's worth mentioning that there is TeaVM which compiles Java bytecode to JavaScript. I have never tried it, but it seems very promising.

Given a set of Java source code files, how can I compile them into one or more JavaScript files that can be used with hand-crafted JavaScript?
Although there are many solutions to convert Java applications to Javascript, you are interested on a solution where new javascript code may interact with the resulting code. This is an update (as 2018) of the other answers.
There are different types of tools. For instance, you may find tools that allow you (1) convert java code to javascript; (2) convert bytecode to javascript, asm.js or webassembly; (3) execute java applications directly in the browser and (4) create solutions that combine java and javascript. You must select the solution to use depending on your requirements.
Converting Java source code to Javascript
Some solutions take java source code and produce a javascript equivalent version. Usually, these solutions transforms the Java to Javascript, but do not support all the behaviours and libraries of the Java runtime. The resulting code may not support some java standard libraries. Typically, they are used to create HTML application using Java but not for migrating the code. Pros: The resulting solution may include very small files. You can use it to reuse your own business logic classes without considering GUI or platform specific libraries. Cons: it is possible that you cannot use some functionalities of the Java platform. It requires access to the source code.
JSweet converts Java to javascript. It includes API bindings for 1000+ javascript libraries. You can write java code that use these libraries.
It converts java to typescript and later uses the typescript compiler to create the corresponding javascript.
It can produce multiple types of javascript modules and typescript definitions. You can use the resulting code in javascript very easily.
j2s, is the compiler used by the Eclipse RAP platform to translate java code to javascript. It is used there to convert the SWT (GUI) widgets to javascript and HTML. It does not support all the Java standard libraries
It may convert simple Java classes to be used in javascript. You must use the classes considering the OO Java-Javascript simulation performed by j2s.
Converting Javascript bytecode to javascript
These solutions take compiled java code (.class files) and produces equivalent code in javascript, asm.js or webassembly. Considering that the java code may depend on java standard libraries (i.e., the JRE), these solutions typically includes ported and pre-compiled libraries. Pros: you do not need to change anything in your code. You do not need the source code neither. Cons: the resulting solution may require the load of a lot of files.
Bck2Brwsr, a Java VM that may compile ahead-of-time the java bytecode to javascript. It produces a javascript file for each .jar file.
You may use the vm javascript object to load a class into javascript and execute static methods (using vm.loadClass(.., function(class){..}}). There is an example in the documentation for the gradle plugin and the maven task.
TeaVM, is another Java VM that may convert ahead-of-time the code to javascript. In contrast to Bck2Brwsr, it supports threads, produces a single file for all your classes and provide better debugging support.
You may call java from javascript using the javaMethods.get(..).invoke(...) method.
DukeScript, transpile java code and bytecode to javascript using Bck2Brwsr or TeaVM.
It provides maven tasks to compile the java code. The resulting code (and technique to call java from javascript) depends on the used transpiler.
Dragome, transpile java bytecode to javascript.
You can annotate static methods in Java and use it in Javascript: e.g. you can annotate a method with #MethodAlias(alias="windows.method1") and invoke it from javascript using window.method1()
CheerpJ (a commercial product) may run complete java applications using Swing and AWT. It provides a very complete javascript environment that support operating system, thread and network functionalities.
It provides a complete runtime API. You can run a main method using cheerpjRunMain( <class>, <jar> ). You can create objects using cjNew( <class>, <params>...) and invoke static methods using cjCall( <class>,<method>,<params>...). There are many other methods you may consider.
Running Java code in Javascript
DoppioJVM is a complete JVM written in Typescript. Pros: It emulates a lot of elements of the operating system, including filesystems, TTY consoles and threads. Cons: Considering that it is an interpeter, it may result slower than other solutions. (I have not tested it)
DoppioJVM is a JVM written in Typescript
The documentation includes snippets of code to load and run the classes. You can run a static method using jvm.runClass( <class>, [ <args>...], function(response){..}). You can run a Jar file and perform many other tasks.
Create applications combining Java and Javascript
Some other solutions provide, not only the tools for compiling the code, but also frameworks and solutions to create java and javascript solutions. For instance, CheerpJ has complete versions of the Swing and AWT libraries for graphical user interfaces, but they may result very slow. You may replace the user interface by using new HTML versions that run faster on the browser. Pros: You can reuse existing code without changes, mainly some libraries and business logic. You may remove from your solutions libraries that run not efficiently in the browser. Cons: If you wanna keep maintaining your java desktop version, you must deal with different code for the browser.
GWT converts Java code to javascript but uses a different set of libraries for the user interface and client-to-server communications.
The documentation maintains information of differences with the JRE
You may export Java classes to be used by Javascript code.
Dukescript uses the conversion tools mentioned before.
It provides a Knockout4j library that may interact easily with HTM, DOM and additional javascript code.
Dragome not only convert java code but also include means to interact with the HTML/DOM
You may use annotations to bind java objects to HTML elements and to handle the DOM
HTML/Java API is an Apache project that standardize the access to HTML/DOM/javascript from transpiled code
It was donated by Dukescript and other tool providers.
Recommendation
If you wanna reuse few classes created by your own, you may try JSweet. You may create javascript modules (libraries) that you can use easily with javascript and typescript.
If you wanna reuse a medium to large codebase that rely on multiple java libraries, you may try CheerpJ, Dukescript or Dragome. You may reuse large parts of your code and create (gradually) the user interface and client-to-server communications using technologies that are more browser-friendly.
If you wanna run complete java applications without change, you may try CheerpJ. It can run Swing and AWT user interfaces. It also provide an Applet runner.

Here's two other options, things to look into and a third option not converting, just living together.
Java2Javascript
I have been wanting to try this out -- Looks closer to what's been asked. Quoting the web page:
an Eclipse Java to JavaScript compiler plugin and an implementation of JavaScript version of Eclipse Standard Widget Toolkit (SWT) with other common utilities, such as java.lang.* and java.util.*. You can convert your SWT-base Rich Client Platform (RCP) into Rich Internet Application (RIA) by Java2Script Pacemaker.
A limited Java in Javascript experience - You'd need to port your necessary dependencies or find alternatives via tools like jQuery, etc.
DukeScript
As I view DukeScript, it compiles some front-end Javascript and calls Java behind, from the browser's Javascript. It seems more or less a hybrid approach so you can use the Java wealth of libraries from Javascript. I will fall foul of a browser security policy for Java.
A more full-Javascript on Java experience leveraging the Java-runtime. If I wanted that outside the browser environment I'd use Javascript on Java.
Nashhorn
Consider this as an example of using Java's resources as foundation for Javascript: Nashorn and JavaFX, as an example for a rich Javascript operated client. Anyway with a Javascript engine inside Java you're not needing to translate between a Javascript-VM to object-code to a Java-VM quite so much.

Not entirely on-topic, but Kotlin is a 100% Java-compatible language that can compile to JavaScript.
IntelliJ IDEA can automatically convert Java to Kotlin and compile it to run on Node or the browser.

Given a set of Java source code files, how can I compile them into one
or more JavaScript files that can be used with hand-crafted
JavaScript?
There is no direct correlation between both the built-in Java API and Java language features, and those of JavaScript. So any attempt at creating a "converter" is going to be incomplete. As a fundamental example, Java classes don't have a direct corresponding JavaScript idiom.
Whether or not an incomplete conversion tool will work for your use case is impossible to know without the source code.
That said, my suggestion to solving your problem would be to first attempt to use GWT: set up a demo project, drop in the source of your library and call it from the JavaScript side and see what GWT outputs in it's .js file. Some of the other tools suggested by other posters here are definitely worth checking out as well.
If that is fruitful and gets you part of the way, great.
From there, you'll need/want to do the remainder of the conversion by hand. After all, assuming you want the code to actually function correctly, a manual review would definitely be in order. Some unit tests being converted along with it would be ideal as well.
You don't state how large the source of your project is, but if it's small (let's say less than a thousand lines of code), even a complete conversion by hand shouldn't be extremely difficult. If it's much larger than that, I would suggest reviewing if you actually want that as JavaScript code anyway.

This question is from 13 years ago; yes, I know. But I'll put my addition here as this is an excellent repository of answers on the topic, and as it comes up first in a Google search it is likely to have the most eyeballs on it.
I'm a former GWT developer. And agree it is an excellent solution & technology. But also yes as mentioned, the cross compilation part mostly (arguably, completely) focuses on the W in its name (Google WEB Toolkit): the client-side JavaScript part of full stack SPA Web applications, its serialization & communication, it's client/server logic partitioning, et al.
I just stumbled onto something that is relatively new -- a 2014-2015 offshoot of GWT placed into the public domain in 2018 -- that I think is more generally focused and what was being requested in the original post (13 years ago, lol). I'm eager to try it.
J2CL: https://github.com/google/j2cl
I read about it here, which to me was a good & thorough intro of the technology (have a look and let me know if you agree or not):
https://www.infoq.com/news/2019/05/j2cl-java-javascript-transpiler/

Are there command line or library tools for rendering webpages that use JavaScript?

Page-scraping on the Internet has seem to have hit somewhat of a wall for me, as there are more and more sites that are dependent on JavaScript for rendering portions of the screen.
It seems to me that with so many open source layout and JavaScript renderers released (like WebKit, Gecko and Chromium + V8) that someone must have made a tool for downloading a page and rendering its JavaScript without having to run an actual browser. However, I'm not turning up what I'm looking for with my searches - I've found tools like Selenium-rc, but they depend on a running browser. I'm interested in any tool or library which can do one (or both) of the following:
A program that can be run from the command line (*nix) which, given the source of a page, returns the page's source as rendered by some JS engine.
Integrated support in a particular language that allows one to (easily) pass the source of a page to it and returns the page's source as rendered by some JS engine.
I think #1 is preferable in a general sense, but #2 would be more useful if the tool exists in the language I want to work in. Also, I'm not concerned with the particular JS engine - any relatively modern one will do. What is out there?

web kit html to pdf works perfect, it can even produce jpg
http://wkhtmltopdf.googlecode.com

You can look at HTMLUnit. It's main purpose is automatic web testing, but I think it may let you get the rendered page.

Well, there's the DumpRenderTree tool which is used as part of the WebKit test suites. I'm not sure how suitable it is for turning into a standalone tool, but it does what you ask for (render HTML, run JavaScript, and dump its render tree out to disk).

Since JavaScript can do quite a lot of manipulations to the web page's document object model (DOM), it seems like to accurately scrape the content of an arbitrary page, you'd need to not only run a JavaScript engine, you'd also need a complete and accurate DOM representation of the page. That's something you'll only get if you have a real browser engine instantiated. It is possible to use an embedded, not-displayed WebKit or Gecko engine for this, then after a suitable loading delay to allow for script execution, just dump the DOM contents in HTML form.

We used Rhino sometime ago to do some automated testing from Java. It seems it'll do the job for you :)

i think there's an example code for Qt that uses the included WebKit to render a page to a pixmap. from there to a full CLI utility is just defining your needs.
of course, for most screen-scraping need you want the text, not a pixmap... if that's what you want, better check Rhino

There is the Cobra Engine for Java (http://lobobrowser.org/cobra.jsp), which handles Javascript (it also has a renderer, but that is optional). I've never used it, but have heard nice things said about it.

It's very little code to have a WebView render a page without displaying anything, but it has to be a GUI application. They can take command line arguments as well, and hide the window. Using WebKit directly it might be possible in a tool.
Apart from the complicated DOM access in Objective-C WebKit can also inject JavaScript, and together with jQuery that makes for a nice scraping solution. I don't know of any universal application doing that, though.