Are there any libraries or frameworks that provide the functionality of a browser, but do not need to actually render physically onto the screen?
I want to automate navigation on web pages (Mechanize does this, for example), but I want the full browser experience, including Javascript. Thus, I'd like to have a virtual browser of some sort, that I can use to "click on links" programmatically, have DOM elements and JS scripts render within it, and manipulate these elements.
Solution preferably in Python, but I can manage others.
PhantomJS and PyPhantomJS are what I use for tasks like these.
What it is, is a headless WebKit based browser which is fully controllable via JavaScript. There's a C++ implementation (PhantomJS) and a Python one (PyPhantomJS). I prefer the Python one though, because it has a plugin system which allows you to add functionality to the core without actually modifying any code, unlike the C++ one. :)
There is an absolute ton of free software technology now available: take your pick at http://wiki.python.org/moin/WebBrowserProgramming but if you have specific questions join pyjamas-dev on google groups and i'll be happy to give further details, there. brief answer: you can run pywebkitgtk "headless", or you can use xulrunner (via python-hulahop) again using pygtk without actually doing "browserwidget.show()", and there's also pykhtml. also you could use python COM to connect to MSHTML.DLL.
these are all "cheat" methods: using python bindings to a graphical web browser engine without actually firing up the graphical bit. if you really wanted to put some serious hard-core programming in, you could create a "port" of webkit which was not connected to a GUI toolkit: as an experienced webkit programmer i'd put it as around... 2 weeks of full-time effort to make such a "headless" version of webkit.
l.
Looks like http://watin.sourceforge.net/ might be a good way to go.
If you don't have to go pure Python, you could do IronPython since it's a C# project.
take a look at this little doosy on ajaxian
http://ajaxian.com/archives/server-side-rendering-with-yui-on-node-js
It also talks about Aptana Jaxer which I think runs on a headless firefox so is basically the Mozilla browser engine in all it's glory.
There is Kapow. Its pure Java and costs money:
http://kapowtech.com/
And there is Lixto: Its Eclipse based and uses Mozilla Gecko as rendering engine (unless they already changed it to WebKit, as they said they'll do years ago). Its very nice and also costs money:
http://www.lixto.com/?page_id=50
They are both graphical tools where you define the site navigation and what should be extracted by point and click. But you can also write xpath and regular expressions and even JavaScript that runs in the sites context.
I used them both in the lectures web data extraction and applied web data extraction at the technical university Vienna (Lixto is written by the Professor who held the lecture).
HTMLUnit in Java is very good. I think it's only the Java implementations of headless browsers that manage to provide Javascript support.
MaxQ, I read about here, sounds like it might be interesting: "written in Java, generates Jython scripts"
Try HtmlUnit !!!
Related
Firefox has a SpiderMonkey javascript engine. Chrome has V8 javascript engine.
Obviously those engines are a separate products and browsers utilize some kind of interface API to interact with them.
On the other hand programmers longed for a long time for their favorite language in browser. So much so, that we have products like GWT (for java), parenscript (for common lisp), HJScript (for haskell), and i'm sure many other libraries for many other languages that allow programmers stay with their favorite language and generate client side code as well.
The idea is so obvious that i am surprised that there's no implementation of it yet. Why not publish the interface API of browser to language engine and allow web sites to provide custom language engines as downloadable bundles. With current internet speeds 3-4 megabytes one time download is not a problem for majority of applications, even more so for intranet usage.
So where's our pluggable engines ?
You don't need pluggable engines really, just an agreed upon byte-code format. Google is going down that path now with NaCl and PNaCl which is based on LLVM. So any program that compiled down to a safe subset of LLVM byte-code could be run in the browser.
Browser vendors can't even agree on a common video format (see the html5 <video> debate) or on how the document DOM object should look like, and you want them to agree on a whole language interface?
Good luck.
I guess you forgot about applets and embed's. Both offer exactly what you want. And both suck for the very same reason.
We've been down this route in the past.
Older versions of IE supported VBScript as a scripting language in addition to JScript.
The result was a whole load of sites that only worked in IE.
This isn't what the web needs again. As a developer, I may desperately want to write code using my favourite language, but as a user I want to be able to browse all sites on the web without having to worry about which plug-ins I need for any given site, or whether my preferred browser can even use those plug-ins.
This is the problem that Microsoft's Silverlight has had. It might be a marvellous technology, but to the end user, why do I want another plug-in? Silverlight has managed to gain some market share thanks to the sheer power of Microsoft, but really not that much.
Now, if the code reaching the end user is consistent, regardless of the language it's written in, then the language doesn't matter. But this effectively means compiled code (or at least bytecode), which is a whole different kettle of fish to running a scripting language in the browser.
What am I asking for is the ability to download a rendered / processed page via Google Chrome or Firefox I think.
For example, I don't want:
hendry#x201 ~$ w3m -dump http://hello.dabase.com
FAIL
I want:
$ $answer http://hello.dabase.com
Hello World
You should be able to do it using PhantomJS. It is running WebKit without the visuals, but you get the same fast and native supports for JavaScript, HTML/DOM, CSS, SVG, Canvas, and many others.
Disclaimer: I started PhantomJS.
Probably too early, but someone ported V8 to Go-lang, so now you could write your own client that makes use of this powerful combo:
http://bravenewmethod.wordpress.com/2011/03/30/embedding-v8-javascript-engine-and-go/
Looks quite straightforward, doesn't require an ugly Java/Rhino stack and adopts the next big programming language.
It looks similar to the problem http://simile.mit.edu/wiki/Crowbar is trying to solve.
You could use jsdom:- https://github.com/tmpvar/jsdom
I'd build a node driver for it, but it's supposed to work with Rhino etc.
I'd take a look at Rhino.
I'd use the excellent env.js library in conjunction with Rhino to simulate the browser environment as much as is technically possible. Once you've implemented some web spider bootstrap code you should be able to obtain the result you want above.
I'd be interested in other solutions to this though.
I built a PHP / JavaScript website for a customer. Then they asked me to replicate it except as a standalone Mac application. I did this with an app that combined an embedded web server, PHP, and 'WebView' - a Cocoa-ish version of the WebKit web browser that I can embed in a standard app window. This all worked great - I got to reuse 10,000+ lines of PHP/JS code, which saved months off of re-implementing it all again in 'native' code.
Now they want a Windows equivalent. I'm reasonably confident I can get PHP and the web server to work. And I know embedding basic IE functionality is pretty easy.
However...in my Mac setup, WebView (via the windowScriptObject stuff) gave me the ability to call JavaScript methods from C++. For instance, I might call a JavaScript method from C++ to update the screen. Likewise I could set things up so that a JavaScript call in the browser could trigger a C++ method - I used this, for instance, to let a user click 'BROWSE' and pick a file path using a real, standard file browser.
So my question is, is there a Windows-based embedded browser setup that would let me interact with JavaScript in this same way?
(the JavaScript <--> WebKit interface is described in much better detail here: http://lipidity.com/apple/javascript-cocoa-webkit/)
Maybe try using something like Appcelerator Titanium so you'll be ready when your client says they want it to work on Linux, or iPhone, or Android.
Quoting Wikipedia: "Appcelerator Titanium is a platform
for developing mobile and desktop
applications using web technologies.
[...] Support for standards-based web
technologies: HTML, CSS and Javascript
on all platforms along with PHP,
Python and Ruby for desktop platforms.
Integrated support for popular
JavaScript and AJAX Frameworks
including jQuery, YUI, MooTools,
Scriptaculous and others."
Sounds like a perfect tool for the job.
When you embed the Web Browser Control (IE), your application code can simply call execScript (http://msdn.microsoft.com/en-us/library/ms536420(v=vs.85).aspx) on the window object. You can have your script call out to the application by using the window.external object from the script, and by using the ObjectForScripting (or C++ equivalent) from the application.
maybe Qt will be good for your case, also you have QtScript and can inject javascript with evaluateJavaScript
I found a great example on the web for invoking JS in my embedded browser from C...basically using COM-ish methods that let you get a DISPID from a script object, and then using the Invoke() method with that. This works great.
But it turns out I need to also call C++ funcs from my JS code. It appears this is possible, and after hours of messing around I think I almost had it - it's like the above in reverse - you create a COM object, then hook it to the browser's script object - but in the end I could not close the deal - I kept getting "library not registered" errors. Honestly I don't know COM well enough to do this right.
So then I, for the heck of it, tried building my first C# app. In about 20 minutes I had an app running with a browser where I could both invoke JS inside of it and have the browser invoke C# methods. Geesh. I'm a believer in .NET after this experience, and a confirmed non-believer in 90's Microsoft technology.
In the interest of completeness, I'd like to mention my Windows port of WebKit, which includes the various cross-layer features of WebKit on the Mac.
I posted some example code showing how to embed WebKit in a native WinAPI application, complete with JavaScript->C++ and C++->JavaScript examples.
The example is a tiny test case for a much larger application using embedded WebViews for major UI components. I can confirm that what you are doing is not only possible, but a great way to build an application.
I'm attempting to automate login into my small community bank, navigating through several links, and then downloading the .OFX.
I can't seem to find a good open source tool which is robust enough to handle this case. Automation frameworks like mechanize can't handle the Javascript.
I need this to be run in a graphic-less environment (via cron), so I don't think I can depend on tools such as watir which control standard browsers.
Any language is fine, although a scripting language is preferred.
Suggestions? Can I tap into the browser engines without displaying the browser?
Take a look at WWW::HtmlUnit. It is a Perl module that uses a Java library (included in the Perl module) that makes it easy to automate web pages, even those with JavaScript.
Use webdriver . The API page says you can toggle the visibility.
If webdriver doesn't fit you, you can use COM directly to automate Internet Explorer, you'd just have to read the documentation page on MSDN.
I'm not really sure how they will react to the graphic-less environment though.
Another option would be to use HtmlUnit, which can handle JavaScript, and emulates a browser, so you don't need to worry about interfacing with a real browser.
As for the scripting part, you can use groovy which compiles to bytecode. If you know Ruby, you'll have no trouble picking up groovy. Here's a link that will help PLEAC groovy
This may do what you want: Test Run: Web UI Automation with Windows PowerShell:
Dr. James McCaffrey - March 2008 Here
we show you how to use Windows
PowerShell to create quick and easy UI
test automation for ASP.NET and
classic ASP Web applications.
For a while now, I have been using UltraEdit on my Windows box. The ability to write scripts with a familiar language (JavaScript) has proved to be extremely useful. The only problem is that I cannot use it on my Linux box at work. Is there a comparable text editor that runs on Linux and has an integrated scripting engine?
Not breaking the bank and being cross-platform would be great.
EDIT:While recordable macros are great, I use the scripting engine much more.
All of the major open-source editors and most of the others hava a scripting facility of some description - some (Emacs in particular) are famous for it. The only ones that don't tend to be very lightweight ones like pico.
vim has a native scripting language and can also be built with embedded Python, Tcl or Perl interepreters that can operate on selections, buffers etc through the plugin mechanism. Emacs is all about scripting - it's has a LISP interpreter built right into the core of the system and most of the editor is written in LISP. There is a running joke about emacs describing it as a LISP interpreter that someone just happened to use to write a text editor.
Vim's user interface is descended from vi, which is somewhat quirky but very powerful once you get used to it. It also does recorded keyboard macros particularly well and has a very nice regular expression search/replace facility.
Emacs is regarded as a bit of a baroque monstrosity and is very large and complex. However, its scripting capability is second to none and there is an enormous variety of macro packages that do many things. It has a very loyal following of people who swear by it; once you've gotten over the learning curve (there is an enormous body of resources on the web to help with this) it's a very powerful system indeed. You can customise emacs into a whole IDE and there are people around who claim to spend the majority of their tube time in it.
Both of these editors can work in text mode or with a GUI and are highly portable, running on a wide variety of platforms. They are both open-source.
I've used both; I used to use XEmacs (a major code-fork of emacs that goes back a number of years) back in the 1990s but went to vim later on. I even use vim on Windows.
If you find the user interface of Vim or Emacs a bit too much, there are a variety of other text editors available, many of which offer scripting. Examples of these are SciTE, which has a built in Lua interpreter, NEdit, which has a homebrew macro language of its own or GEdit, which is substantially written in Python (which can also be used for scripting it) and has a plugin API.
EDIT: Outside of a few specific projects (e.g. Mozilla) Javascript never got much traction as a stand-alone or embedded scripting language in open-source circles. Historically there wasn't a popular open-source Javascript interpreter that got widespread acceptance in the way that Python or Tcl/Tk did. Javascript is more widely used in closed source systems such as UltraEdit or InDesign (to name a couple) whereas other languages were more popular on open-source projects.
None of the open-source text editors that I am aware of feature javascript as an option for a scripting language (feel free to step in and comment or edit this if you know of one). You will probably have to move off Javascript to another language such as Python or LISP. However, now that QT comes with a Javascript interpreter (QTScript) you may find some of the KDE-based ones offering this as a scripting option, but I am not specifically aware of any off the top of my head.
emacs is free and has its own embedded lisp dialect which can be used to write nearly anything, including light scripting as well as mail user agents and IRC clients ;-)
There's a bit of a learning curve, but my experience with emacs has been very positive. I don't like modal interfaces too much, and no other editor puts the navigation shortcuts right under your fingers.
Vim is omnipresent and vimscript is really easy and text-editing oriented: http://vimdoc.sourceforge.net/htmldoc/usr_41.html
That comes by default. You can also use Python, Ruby, Perl, Scheme... but that requires compiling.
Python might be a good choice, since omnicomplete requires it. I run a separate binary for omnicomplete anyway since I like the ultra-lean vim for other usage, check here:
http://vim.wikia.com/wiki/Compile_a_separate_copy_of_Vim_for_Python_coding
Honestly I haven't used python to script vim per se. I usually code, python or not, with vanilla vim and no auto-complete. I do however call python and any other command from vim to process my files line by line, might reply with more about that if you are interested, but that doesn't require to compile anything, works as is.
EDIT: if you want a point-and-click editor, you can run Cream on gVim http://cream.sourceforge.net/download.html
I still think that vim takes little time to get used to and is VERY worth learning, more so considering you can use your basic vi knowledge to connect to a server via SSH no problem, and it's installed in any *nix. I like knowledge I can reuse, saves me time and trouble long term.
I use jEdit, that is a great editor and allows to be scripted with beanshell. As it is written in Java it runs well under Windows and Linux.
Simon Groenewolt mentioned, that a plugin (JavascriptShell) exists, that allows you to write Macros and Scripts also in Javascript, not only in Beanshell.
If you don't need really complex scripting, vim/gvim allows you to record a keystroke sequence and play it back. And you can give a repeat count on the playback, so you can record an operation on one line, then repeat it for the next 10,000 lines in one step.
Wow, I really don't want to start a holy war here, but all these Emacs recommendations are kind of missing the point. Emacs is very powerful, but let's be honest. It doesn't have a learning curve so much as a learning WALL. Going from a point-and-click IDE style editor like UltraEdit is going to be a huge culture shock.
And honestly, when someone says they like to write scripts in javascript, and are looking for something similar in Linux, the first thing you say is "learn LISP"?!
This is why people think Linux is hard to use
I'm going to go in another direction entirely and suggest Aptana Studio
It has built-in highlighting and code-completion for a variety of languages, and supports scripting in javascript with Eclipse Monkey. It is based on Eclipse, and therefore runs on java hence multiplatform. And it is available for free.
It's kind of cliche, but emacs. Or am I misunderstanding what UE's script engine is?
Try Emacs, either XEmacs or GNU Emacs.
I use gedit. You can write plugins in Python.
Komodo Edit is made on top of Mozilla's XUL Runner. It offers possibilities for extending it thorough either extensions (like Firefox) or macros, snippets, commands. You may write these snippets in either JavaScript or Python, which is nice, but you still have to know the API in order to do something useful.
Being built with Mozilla technology it runs on Linux too.
It looks like Komodo Edit, SciTE, and Eclipse Monkey are the winners. Komodo Edit seems to be most similar to Ultra Edit. SciTE is something I've used before, and Lua is not that difficult; SciTE's API though does not seem as extensive as Komodo Edit's API. Eclipse Monkey is something I am definitely going to use, but it requires Eclipse, which is definitely not a text editor.
EDIT: UltraEdit is coming out for Mac and Linux Soon.
The traditional way for doing scripted text editing in Linux is to use the facilities that have (almost) always been available in *nixes: sed, awk, grep, things of that nature. Sure, they maybe don't appear to be as "handy" as one might find an integrated Javascript engine, but they are very mature and work well. If this scripting language MUST be inside the editor, Emacs is probably a good one here too.
You could use an "integrated" type solution to solve the problem, as lots of others have mentioned, but IMHO the commandline is more powerful in this regard.
SciTE can be scripted with Lua and is a good, simple editor which behaves much the same way if you're switching between Linux and Windows every day.
I don't know what you're using the scripting capabilities in your editor for, but you may want to consider automating those tasks using Linux command-line tools such as sed or awk.