Retrieving a javascript processed Web page - javascript

What am I asking for is the ability to download a rendered / processed page via Google Chrome or Firefox I think.
For example, I don't want:
hendry#x201 ~$ w3m -dump http://hello.dabase.com
FAIL
I want:
$ $answer http://hello.dabase.com
Hello World

You should be able to do it using PhantomJS. It is running WebKit without the visuals, but you get the same fast and native supports for JavaScript, HTML/DOM, CSS, SVG, Canvas, and many others.
Disclaimer: I started PhantomJS.

Probably too early, but someone ported V8 to Go-lang, so now you could write your own client that makes use of this powerful combo:
http://bravenewmethod.wordpress.com/2011/03/30/embedding-v8-javascript-engine-and-go/
Looks quite straightforward, doesn't require an ugly Java/Rhino stack and adopts the next big programming language.

It looks similar to the problem http://simile.mit.edu/wiki/Crowbar is trying to solve.

You could use jsdom:- https://github.com/tmpvar/jsdom
I'd build a node driver for it, but it's supposed to work with Rhino etc.

I'd take a look at Rhino.
I'd use the excellent env.js library in conjunction with Rhino to simulate the browser environment as much as is technically possible. Once you've implemented some web spider bootstrap code you should be able to obtain the result you want above.
I'd be interested in other solutions to this though.

Related

Web-based JavaScript Debugger

I have an idea for a project that has to do with helping people learn to program. One of the things I think would be cool would be to have a place on the site where you can edit JavaScript code in real time in your browser.
I know there are a number of web-based JavaScript editors, but I haven't been able to find out if there are any tools that let you debug JavaScript from your browser. I think being able to step through your code is a valuable learning experience for people who are new to programming. It seems like something along those lines should be possible, though.
I originally thought you could use something like Firebug Lite: http://getfirebug.com/firebuglite, but they don't support debugging. Is this just too hard of a problem to solve in JavaScript?
I suggest you try Cloud9. It is a full-featured web based IDE for Javascript, which includes a great debugger. It is free for open source projects, which I imagine would include most learning purposes.
Why not just use the build in Javascript console from the browser? Every modern browser has this feature.

What's the best way to do integration testing for a Javascript heavy UI in a rails app?

We have a web application that makes extensive use of AJAXy Javascript in the UI. We have nearly complete code coverage of our backend using Shoulda and Webrat, and would like to extend our test suite to include full integration testing through the Javascript UI.
We tried Selenium but found it brittle and temperamental. Are there more reliable options?
UPDATE
For those still checking this out, we ended up using Xvfb so we can run Firefox without a screen. Allows us to run the test on a headless Jenkins CI server. We still have to run tests "live" locally occasionally to debug, but it works pretty well.
One of the JavaScript gurus where I work recently pointed out PhantomJS as an interesting tool for testing our JavaScript-heavy web applications. We haven't tried it out yet but the idea of a headless WebKit for DOM testing sounds promising to me.
This is something I have been wrestling with for a while, as I am doing some work with ExtJS (a very powerful JavaScript UI builder for the browser) and Rails.
After having researched quite a few different options. I still haven't found a perfect solution for it. Ideally, I would be able to run them headless and just report on the output. Unfortunately, none of the emulators out there seem to be able to run JavaScript with full DOM support seamlessly (at least, none of the options I've found are). So that pretty much means that you have to run your full-powered JavaScript code in a real interpreter (such as a browser). Webrat with Selenium works acceptably well, assuming you're willing to deal with the pain of trying to path out your requests to the UI properly. If it's your own JavaScript that you're implementing it against, that may be easier. But when it comes to a third party UI library that you don't have much control over, it can certainly get, shall we say, interesting.
Probably not the most helpful response, but that has been my findings up to now!
Hmm I would give Capybara a look, it can use selenium-webdriver (not to be confused with selenium-RC, they are different) for javascript testing. I haven't found it very brittle when compared with Webrat... it seems to be fairly consistent.
As Chris Rueber says, there aren't really any headless DOM interpreters that support JS as well - for now it's fire up a web browser for your automation or write unit tests in the javascript itself (Which isn't really integration testing either).
When you have a lot of selenium-webdriver-backed tests they can take awhile to run sometimes, but it's surely better than no tests at all.
check out the gem jasminerice to test your js logic.
https://github.com/bradphelan/jasminerice
for the integration test I would recommend to use rspec with capybara as acceptance tests. distinguish request specs and acceptance specs!
another possibility is to use turnip as an alternative to cucumber.
https://github.com/jnicklas/turnip
to speed up your tests test headless. You could use capybara-webkit (depends on qt) or poltergeist (which depends on phantomjs).
both are easily to set up. I prefer poltergeist.
There are a couple of gems you could use if you didn't like Selenium.
The one I recommend is Jasmine: https://github.com/pivotal/jasmine
You can also check out Culerity: https://github.com/langalex/culerity

Javascript (and HTML rendering) engine without a GUI for automation?

Are there any libraries or frameworks that provide the functionality of a browser, but do not need to actually render physically onto the screen?
I want to automate navigation on web pages (Mechanize does this, for example), but I want the full browser experience, including Javascript. Thus, I'd like to have a virtual browser of some sort, that I can use to "click on links" programmatically, have DOM elements and JS scripts render within it, and manipulate these elements.
Solution preferably in Python, but I can manage others.
PhantomJS and PyPhantomJS are what I use for tasks like these.
What it is, is a headless WebKit based browser which is fully controllable via JavaScript. There's a C++ implementation (PhantomJS) and a Python one (PyPhantomJS). I prefer the Python one though, because it has a plugin system which allows you to add functionality to the core without actually modifying any code, unlike the C++ one. :)
There is an absolute ton of free software technology now available: take your pick at http://wiki.python.org/moin/WebBrowserProgramming but if you have specific questions join pyjamas-dev on google groups and i'll be happy to give further details, there. brief answer: you can run pywebkitgtk "headless", or you can use xulrunner (via python-hulahop) again using pygtk without actually doing "browserwidget.show()", and there's also pykhtml. also you could use python COM to connect to MSHTML.DLL.
these are all "cheat" methods: using python bindings to a graphical web browser engine without actually firing up the graphical bit. if you really wanted to put some serious hard-core programming in, you could create a "port" of webkit which was not connected to a GUI toolkit: as an experienced webkit programmer i'd put it as around... 2 weeks of full-time effort to make such a "headless" version of webkit.
l.
Looks like http://watin.sourceforge.net/ might be a good way to go.
If you don't have to go pure Python, you could do IronPython since it's a C# project.
take a look at this little doosy on ajaxian
http://ajaxian.com/archives/server-side-rendering-with-yui-on-node-js
It also talks about Aptana Jaxer which I think runs on a headless firefox so is basically the Mozilla browser engine in all it's glory.
There is Kapow. Its pure Java and costs money:
http://kapowtech.com/
And there is Lixto: Its Eclipse based and uses Mozilla Gecko as rendering engine (unless they already changed it to WebKit, as they said they'll do years ago). Its very nice and also costs money:
http://www.lixto.com/?page_id=50
They are both graphical tools where you define the site navigation and what should be extracted by point and click. But you can also write xpath and regular expressions and even JavaScript that runs in the sites context.
I used them both in the lectures web data extraction and applied web data extraction at the technical university Vienna (Lixto is written by the Professor who held the lecture).
HTMLUnit in Java is very good. I think it's only the Java implementations of headless browsers that manage to provide Javascript support.
MaxQ, I read about here, sounds like it might be interesting: "written in Java, generates Jython scripts"
Try HtmlUnit !!!

Web Automation Tool

I've realized I need a full-fledged browser automation tool for testing user interactions with our JavaScript widget library. I was using qunit, starting with unit testing and then I unwisely started incorporating more and more functional tests. That was a bad idea: trying to simulate a lot of user actions with JavaScript. The timing issues have gotten out of control and have made the suite too brittle. Now I spend more time fixing the tests, then I do developing.
Is it possible to find a browser automation tool that works in:
Windows XP: IE6,7,8, FF3
OSX: Safari, FF3
?
I've looked into SeleniumIDE and RC, but there seems to be some IE8 problems.
I've also seen some things about Google's WebDriver, which confusingly seems to work with Selenium.
Our organziation has licenses for IBM's Rational Functional Tester, but I don' think that will work on the MAC.
The idea is to try to run tests on all the browsers our organization supports. Doable? Are my requirements unrealistic? Any recommendations as far as software to try?
Thanks!
I would recommend using Selenium but I say that as a Selenium Committer.
Selenium works on any browser that supports JavaScript since the framework has been written in JavaScript. This means if your browser on any OS supports JavaScript it will run in Selenium. That documentation it out of date, you can see that since it is talking about IE8b1 and IE9 preview is out now.
Selenium and WebDriver (which isn't a Google thing since it started at ThoughtWorks) are currently being merged as they both have their strengths and weaknesses. The current merged work will be called Selenium 2 and you can start using the alpha release now at http://code.google.com/p/selenium/. It will still work on any OS as that is still the main driving force behind the work being done.
Selenium IDE only works on Firefox because it is a Firefox add on.
I personally would avoid Rational Functional Tester because it has a lot of weaknesses that its not even worth contemplating.
If you start with Selenium there are some tutorials on my site at http://www.theautomatedtester.co.uk
Try Sahi (http://sahi.co.in/) It works across browsers and operating systems. It has a powerful recorder, and great APIs for object identification. It supports HTTPS, proxy tunneling etc. and has drivers in sahi script, java and ruby. It also has parallel playback inbuilt. It is 5 yr old mature project hosted on SourceForge, with releases almost every month.
It automatically waits for AJAX and page loads, and does not use XPaths for object identification. It also handles sites with dynamic ids.
Selenium is probably your best bet out of the tools you mentioned. What are the issues it has with IE8? You might want to check out HttpUnit to see if that meets your needs, also.
Selenium RC is a great tool if you invest the time to use it. With significant modifications to the existing library I've gotten it to fulfill all of my front end testing needs.
The confusion you are having about Webdriver is understandable. Selenium 2 is in development and will be a merge of Webdriver and Selenium. Check out: http://www.youtube.com/watch?v=RQD4EzWI4qk to get more detail.
The only browser that I have found to be unusable with Selenium is IE6. IE7 and IE8 work fine as does Firefox (which I have modified to include firebug for debugging purposes).
I'm in the same boat. It is a difficult problem to solve. Windmill and Selenium are the 2 best I've found. Though they both have issues. Selenium can only record scripts in Firefox and I haven't managed to get the proxy chaining to work as advertised. Windmill you can record in any browser and you can supposedly tweak the proxy to put extra logic in there, but the js mechanism for recording across page loads has been in my experience very brittle at least on the app I have to test.
I don't think anyone can get it quite right as long as there is more than one browser that needs to be supported.
Maybe have a look at SIKULI. It's a different paradigm but, depending on what you want to test exactly, it may do the job and will work with any browser, on any platform.
Have a look at their official blog for some examples of interactions with web applications.
So I wrote some of my more problematic tests in Selenium RC, using the Python driver. It was a better experience than writing the same tests in pure JavaScript, but I still had some of the same issues.
Testing something like an ajax autocomplete widget, meant forking some of the code depending on IE, or Firefox, and I still can't get typeKeys or a combination of type with typeKeys to work in Safari.
So, I am not sure if having cross browser clean, extensive ui tests is a bit unrealistic.
Should I try webdriver/Selenium 2? Would that make things better, or is that product not ready for prime time yet? How's the Python binding for that? I don't know Java, but I would learn some if need be.

Breakpoints in Ruby IDE?

I currently use TextMate for Ruby/Javascript/Actionscript development and it is amazing. But one thing I would really love to use are breakpoints so I could stop code execution and examine the state of the variables and walk through the code. Something like what Flex Builder does.
Does TextMate have this capability? Or what do you use to do breakpoints and that sort of thing with Ruby? How about for Javascript too?
Thanks!
Since TextMate is not an IDE but just a text editor (on steroids though!) I believe this is something totally not supported.
As for the javascript, this is something you can do with firebug (at least for firefox) and similar tools are included (or can be found) for IE, Chrome and Safari :)
There are 3 mostly-Java IDEs that run under Mac OS X yet do a good job of developing (and of course debugging) Ruby.
Eclipse, with the Aptana plugin;
IntelliJ IDEA (it knows lots of programming languages)
NetBeans (I think).
All three are free (even IntelliJ, they have an Open Source edition out now), so you can just download, play around with it, get accustomed and go to town.
Eclipse with the Dynamic Languages Toolkit supports Ruby debugging. I have used it in the past, and it supports Javascript as well.
TextMate is an editor and not a full-blown IDE, so it doesn't support debugging.
If you're looking for a Ruby IDE, I highly recommend RubyMine. It's got great support for debugging, running tests, easily navigating between files, and basic refactoring. Admittedly it's a lot more heavyweight, so I tend to use it for major work (or when getting up to speed on an existing code base), and still use TextMate for quick changes.
There are some open source solutions as well. In the past I've used NetBeans and RadRails, and particularly NetBeans has worked very well for me, too. Not as good as RubyMine, but well worth checking out if you're on a budget.
I should also point out that you can debug Ruby apps from the command line, using the ruby-debug gem. Basically, you place a debugger call into your code to set a breakpoint, and then run the app with rdebug instead of ruby. Check out this article for an overview.
arcadia is a ruby editor written in ruby with ruby debugging support. A little rough still, but at least you get the good feeling of running ruby :)
-r

Categories