I'm going to write bunch of browser extensions (the same functionality for each popular browser). I hope, that some of the code will be shared, but I'm not sure about this yet. For sure some of extensions will use native API. I have not much experience with TDD/BDD, and I thought it's good time to start folowing these ideas from this project.
The problem is, I have no idea how to handle it. Should I write different tests for each browser? How far should I go with these tests? These extensions will be quite simple - some data in a local storage, refreshing a page and listening through web sockets.
And my observation about why is it hard for me - because there is a lot of behaviour, and not so much models, which are also dependent on a platform.
I practise two different ways of testing my browser extensions:
Unit tests
Integration test
Introduction
I will use the cross-browser YouTube Lyrics by Rob W extension as an example throughout this answer. The core of this extension is written in JavaScript and organized with AMD modules. A build script generates the extension files for each browser. With r.js, I streamline the inclusion of browser-specific modules, such as the one for cross-origin HTTP requests and persistent storage (for preferences), and a module with tons of polyfills for IE.
The extension inserts a panel with lyrics for the currently played song on YouTube, Grooveshark and Spotify. I have no control over these third-party sites, so I need an automated way to verify that the extension still works well.
Workflow
During development:
Implement / edit feature, and write a unit test if the feature is not trivial.
Run all unit tests to see if anything broke. If anything is wrong, go back to 1.
Commit to git.
Before release:
Run all unit tests to verify that the individual modules is still working.
Run all integration tests to verify that the extension as whole is still working.
Bump versions, build extensions.
Upload update to the official extension galleries and my website (Safari and IE extensions have to be hosted by yourself) and commit to git.
Unit testing
I use mocha + expect.js to write tests. I don't test every method for each module, just the ones that matter. For instance:
The DOM parsing method. Most DOM parsing methods in the wild (including jQuery) are flawed: Any external resources are loaded and JavaScript is executed.
I verify that the DOM parsing method correctly parses DOM without negative side effects.
The preference module: I verify that data can be saved and returned.
My extension fetches lyrics from external sources. These sources are defined in separate modules. These definitions are recognized and used by the InfoProvider module, which takes a query, (black box), and outputs the search results.
First I test whether the InfoProvider module functions correctly.
Then, for each of the 17 sources, I pass a pre-defined query to the source (with InfoProvider) and verify that the results are expected:
The query succeeds
The returned song title matches (by applying a word similarity algorithm)
The length of the returned lyrics fall inside the expected range.
Whether the UI is not obviously broken, e.g. by clicking on the Close button.
These tests can be run directly from a local server, or within a browser extension. The advantage of the local server is that you can edit the test and refresh the browser to see the results. If all of these tests pass, I run the tests from the browser extension.
By passing an extra parameter debug to my build script, the unit tests are bundled with my extension.
Running the tests within a web page is not sufficient, because the extension's environment may differ from the normal page. For instance, in an Opera 12 extension, there's no global location object.
Remark: I don't include the tests in the release build. Most users don't take the efforts to report and investigate bugs, they will just give a low rating and say something like "Doesn't work". Make sure that your extension functions without obvious bugs before shipping it.
Summary
View modules as black boxes. You don't care what's inside, as long as the output matches is expected or a given input.
Start with testing the critical parts of your extension.
Make sure that the tests can be build and run easily, possibly in a non-extension environment.
Don't forget to run the tests within the extension's execution context, to ensure that there's no constraint or unexpected condition inside the extension's context which break your code.
Integration testing
I use Selenium 2 to test whether my extension still works on YouTube, Grooveshark (3x) and Spotify.
Initially, I just used the Selenium IDE to record tests and see if it worked. That went well, until I needed more flexibility: I wanted to conditionally run a test depending on whether the test account was logged in or not. That's not possible with the default Selenium IDE (it's said to be possible with the FlowControl plugin - I haven't tried).
The Selenium IDE offers an option to export the existing tests in other formats, including JUnit 4 tests (Java). Unfortunately, this result wasn't satisfying. Many commands were not recognized.
So, I abandoned the Selenium IDE, and switched to Selenium.
Note that when you search for "Selenium", you will find information about Selenium RC (Selenium 1) and Selenium WebDriver (Selenium 2). The first is the old and deprecated, the latter (Selenium WebDriver) should be used for new projects.
Once you discovered how the documentation works, it's quite easy to use.
I prefer the documentation at the project page, because it's generally concise (the wiki) and complete (the Java docs).
If you want to get started quickly, read the Getting Started wiki page. If you've got spare time, look through the documentation at SeleniumHQ, in particular the Selenium WebDriver and WebDriver: Advanced Usage.
Selenium Grid is also worth reading. This feature allows you to distribute tests across different (virtual) machines. Great if you want to test your extension in IE8, 9 and 10, simultaneously (to run multiple versions of Internet Explorer, you need virtualization).
Automating tests is nice. What's more nice? Automating installation of extensions!
The ChromeDriver and FirefoxDriver support the installation of extensions, as seen in this example.
For the SafariDriver, I've written two classes to install a custom Safari extension. I've published it and sent in a PR to Selenium, so it might be available to everyone in the future: https://github.com/SeleniumHQ/selenium/pull/87
The OperaDriver does not support installation of custom extensions (technically, it should be possible though).
Note that with the advent of Chromium-powered Opera, the old OperaDriver doesn't work any more.
There's an Internet Explorer Driver, and this one does definitely not allow one to install a custom extension. Internet Explorer doesn't have built-in support for extensions. Extensions are installed through MSI or EXE installers, which are not even integrated in Internet Explorer. So, in order to automatically install your extension in IE, you need to be able to silently run an installer which installs your IE plugin. I haven't tried this yet.
Testing browser extensions posed some difficulty for me as well, but I've settled on implementing tests in a few different areas that I can invoke simultaneously from browsers driven by Selenium.
The steps I use are:
First, I write test code integrated into the extension code that can be activated by simply going to a specific URL. When the extension sees that URL, it begins running the tests.
Then, in the page that activates the testing in the extension I execute server-side tests to be sure the API performs, and record and log issues there. I record the methods invoked, the time they took, and any errors. So I can see the method the extension invoked, the web performance, the business logic performance, and the database performance.
Lastly, I automatically invoke browsers to point at that specific URL and record their performance along with other test information, errors, etc on any given client system using Selenium:
http://docs.seleniumhq.org/
This way I can break down the tests in terms of browser, extension, server, application, and database and link them all together according to specific test sets. It takes a bit of work to put it all together, but once its done you can have a very nice extension testing framework.
Typically for cross-browser extension development in order to maintain a single code-base I use crossrider, but you can do this with any framework or with native extensions as you wish, Selenium won't care, it is just driving the extension to a particular page and allowing you to interact and perform tests.
One nice thing about this approach is you can use it for live users as well. If you are providing support for your extension, have a user go to your test url and immediately you will see the extension and server-side performance. You won't get the Selenium tests of course, but you will capture a lot of issues this way - very useful when you are coding against a variety of browsers and browser versions.
Related
How to run compiled files directly using Google Native Client (PNaCl)? It tried checking their documentation. It said that -
Native Client is a sandbox for running compiled C and C++ code in the browser efficiently and securely, independent of the user’s operating system.
But in their documentation, they only deal with sources of the application. Is there any way to run compiled code directly? I want to run files with .exe and .deb extensions
I'm not limiting the answer to Native Client. Any mechanism which can do that sort of work will work for me.
You can't run pre-compiled code within NaCl or PNaCl. You have to use the compilers provided by the SDK. There are three main reasons for this:
NaCl is an execution sandbox which relies on crafting machine code (x86-32, x86-64, ARM, MIPS) in a very particular way. This is regular machine code from the CPU's point of view, but allows the sandbox to run a validator and make sure that the code can't do anything malicious. This is called Software Fault Isolation, and is explained in this paper. The other ISA sandboxes are also documented.
PNaCl targets NaCl, but is an architecture-agnostic intermediate representation. This means that you ship what can be thought of as bytecode, and the browser figures out which type of machine code (x86-32, x86-64, ARM, MIPS) to generate based on the user's machine. The developer doesn't generate 4 binaries.
In both above cases, the code can execute as-is on Windows, MacOSX, Linux, ChromeOS, and (while not usually shipping) Android. This means that the NaCl sandbox presents itself as an operating system, and offers the same APIs. These APIs are different from other OSes, though they're pretty close to POSIX especially if you use nacl_io.
The above points require that you use the compilers provided by the SDK.
It is technically possible to run binaries built for other architectures or operating systems since the system is Turing-complete. That's what QEMU does, what Rosetta did, what Transmeta did, and what the Android Runtime for Chome (ARC) enables. This usually requires binary translation and emulation of all operating system calls. This is technically difficult to implement, and often has severe performance cost. I do not recommend exploring this option.
As #JFBastien pointed out, emulation is the only option to execute pre-compiled native code in the browser environment. But it is an option nevertheless. Depending on your demands on performance, it might even be a viable option.
Click here for example to boot up an emulator running Windows (a very old version though) in your browser.
From the menu pick, for example, notepad.exe (using the cursor down key on your keyboard) and hit enter. There you have it: an unmodified, precompiled, native notepad.exe running inside your browser! (and probably even faster than back in the day when this OS was new).
There are a lot of emulators written in Javascript all around the web. Running a small Linux distribution with usable performance and even with networking(!), graphics and sound is actually possible. Check out the OpenRISC emulator. You can even run an ssh daemon and log into it from your local machine!
Are there any headless browsers for node.js that support dumping a rendered page out to a file? I know phantomjs supports rendering to a file, but it doesn't run on node.js. I know zombie.js is a node.js headless browser, but it doesn't support rendering to a file.
I doubt you will find anything that is going to work as well as phantomjs. I would just treat the rendering as an async backend process and execute phantom in a subprocess from your main node.js process and call it a day. Rendering a web page is HARD, and since phantom is based on WebKit, it can actually do it. I don't think there will ever be a node library that can render a web page to a graphic file that isn't built upon an existing browser rendering engine. But maybe one day phantomjs will integrate more seamlessly with node.
Try nightmare, it uses the electron, it is way faster than phantomjs, and it's API easy and uses modern ES6 javascript.
This might look like a solution with a little bit overhead...
You can use the Mozilla Firefox with the MozRepl plugin. Basically this plugin gives you a telnet port to your Firefox which allows you to control the browser from the outside. You can open URLs, take screenshots, etc.
Running the Firefox with the Xvfb server will run it in headless mode.
Now you just have to control the browser from the outside with node.js. I've seen a few examples where someone has implemented a http alike interface inside the chrome.js of Firefox. So you can run a http command to get a screenshot. You can then use http calls from node.js. This might look strange, it actually is but might work well for you.
http://hyperstruct.net/2009/02/05/turning-firefox-into-a-screenshot-server-with-mozrepl/
I'm running a slightly modified version in production with Perl Mojolicious in async mode to trigger the screenshots. However, there is a small problem. When plugins are required they do work, however Flash usually gets activated when it's in the visible area, this won't happen so movies/flash things might not get initialized.
You might find this helpful, though it's not javascript specific.
There is a webkit-based tool called "wkhtmltopdf" that I understand includes javascript support using the QT web-kit widget. It outputs a visual representation ("screenshot" if you will) of the page in PDF format.
FWIW, there are also PHP bindings for it here: php-wkthmltox
The Chrome dev team has released Puppeteer which can be used in node. It uses Chrome with the headless option.
There's a project called Node-Chimera. Although it's not as mature as Phantomjs, it has all the features you have mentioned: it runs on native Nodejs, and allows you to render pages to a file. Repository is here: https://github.com/deanmao/node-chimera. It has examples to do exactly what you need.
In my web page, I have to start a desktop application on the client's computer if it's installed. Any idea how I can do this?
If the application is MS Office or Adobe Reader, I know how to start them, but the application I want to start is a custom application. You can not find it on the internet.
How can I open the application?
Basically it's not possible to achieve unless an application registers a protocol that will trigger it. If it does that all you need to do is to provide a link using this protocol
yourcustomapp://some.parameters
Another way the 3rd party app can integrate with the browser is if it hooks to it as a plugin. This is how flash apps work etc.
If the app you are trying to launch does not support something like that it's going to be close to impossible to achieve what you want.
The browser sandbox prohibits you from executing local resources, for good reason - to thwart a website destroying your box with malicious code. I've been researching the same functionality.
The only solution I've found is to build an extension in Mozilla Firefox which can launch your app. Extensions live outside the sandbox so they can execute local resources. See this page for how to do that. You may be able to do it cross-browser using crossrider, though I haven't had success with that yet.
You could alternatively build a thick client populated from a web service, and launched from the browser through an extension as mentioned above. This is what I'm doing to get around the sandbox. I'm using local XUL for this.
See my question for additional discussion.
First off - you can't do it using javascript in any sort of a portable mechanism.
If the application is ms office or adobe reader,I know how to startup them
No you don't - you know how to send a document, which the browser associates with these applications and invokes them supplying the name of the local copy of the response. You can't just start the programs.
You just need to do the same for your app - invent a new mime type (the major type would be 'application' and by convention, non-standard minor types are prefixed with 'x-', so you might use application/x-hguser) then associate that mimetype with the relevant program browser side.
i.e: You need to explicitly configure each browser
I already encouter that problem in some complex production environnements.
I do the trick using the following code :
function launch(p_app_path)
{
var oShell = new ActiveXObject("WScript.Shell");
oShell.Run('"' + p_app_path + '"', 1);
}
In IE options > Security > Customize the level > ActiveX controls and plugins > Initialization and script ActiveX controls not marked as safe for scripting, set the value to Ask or Active.
It isn't a security problem when your website is enclosed into a specific security context.
And as they say, it's not worth it to build a gas plant.
JavaScript alone can't do this. (No, not even with MS Office or Adobe Reader.) Thankfully.
There are a number of old ways, including using ActiveX, which may work for your needs. As others have pointed out while typing this, you can customize responses based on the mime type or the protocol, etc.
Any way you look at it, you're going to need control over the end users' browser. If you're in a close environment where you can dictate policy (users must use a specific browser, with a specific configuration), then you're going to need to do that. For an open environment with no control over the end users, you're out of luck.
I'm actually having a lot of success right now with SiteFusion. It's a PHP client/server application framework that serves out XUL/JavaScript applications from a server deamon running in Apache. You access applications from a very thin client in XULRunner, or potentially off a web page using extensions. Clients can execute on any platform, and they're outside of the browser sandbox so you can access local resources such as executables. It'a a fairly elegant solution, their website provides great examples and documentation, and their forum is very responsive. I actually found a minor bug in passing arguments to local executables, posted a question about the forum, and it was fixed by the chief developer in under 15 minutes. Very impressive, overall!
I'm not talking about browser exploits. I'm talking about real applications used in real companies, like Ijji and Nexon.
Basically, from their websites you can click a "Start Game" button, which will launch an executable located at c:\ijji\english or c\nexon[gamename] respectively. These applications are real desktop applications, meaning that they can take advantage of the filesystem, direct3d, and OS [in the form of executing other applications]. The applications can also be launched through command line [as opposed to going to the game host's website].
I figured this would be possible if the application created an ActiveX object to call for the creation of a new process. However, the websites are able to launch applications from multiple browsers other than Internet Explorer, including chrome, which, to my knowledge, does not implement ActiveX.
Obviously the people developing these applications use their own means to do this.
From looking at the services list as well as currently running applications list, I have no indication that they're running something like "gameLaunchingServer.exe" which listens to some obscure port for an incoming connection [to be accessed using iframe - HTTP Protocol] and responds by launching an application...
I'm stumped, and this is sort of stuck in my mind. Obviously, they're not using some random browser exploit, otherwise people at http://www.[insertMaliciousWebsiteHere].com would have jumped on the opportunity already to install random crap. Regardless, it seems pretty cool, and I wanted to know how it worked.
Just curious, hehe.
I believe what they're doing is setting up their own protocol handler on install - when a browser is asked to access an address with a protocol that it doesn't know how to handle (for instance, a steam:// address), it looks at all the installed protocol handlers to find a match.
So you can register your application as a myApplication:// protocol handler, and then your web page can link to a myApplication:// address and launch your application.
I didn't quite find the button you are talking about, but I'm thinking it works only after you installed the application once, isn't it?
In that case, the application probably created its own protocol, just as skype, msn and a bunch of clients.
Having a protocol is the easiest way (and very easy indeed to implement - a simple registry key).
Another way which is used is an extension or plugin.
I thought they were run through plug-ins or like applets.
For example, MS SilverLight
I've realized I need a full-fledged browser automation tool for testing user interactions with our JavaScript widget library. I was using qunit, starting with unit testing and then I unwisely started incorporating more and more functional tests. That was a bad idea: trying to simulate a lot of user actions with JavaScript. The timing issues have gotten out of control and have made the suite too brittle. Now I spend more time fixing the tests, then I do developing.
Is it possible to find a browser automation tool that works in:
Windows XP: IE6,7,8, FF3
OSX: Safari, FF3
?
I've looked into SeleniumIDE and RC, but there seems to be some IE8 problems.
I've also seen some things about Google's WebDriver, which confusingly seems to work with Selenium.
Our organziation has licenses for IBM's Rational Functional Tester, but I don' think that will work on the MAC.
The idea is to try to run tests on all the browsers our organization supports. Doable? Are my requirements unrealistic? Any recommendations as far as software to try?
Thanks!
I would recommend using Selenium but I say that as a Selenium Committer.
Selenium works on any browser that supports JavaScript since the framework has been written in JavaScript. This means if your browser on any OS supports JavaScript it will run in Selenium. That documentation it out of date, you can see that since it is talking about IE8b1 and IE9 preview is out now.
Selenium and WebDriver (which isn't a Google thing since it started at ThoughtWorks) are currently being merged as they both have their strengths and weaknesses. The current merged work will be called Selenium 2 and you can start using the alpha release now at http://code.google.com/p/selenium/. It will still work on any OS as that is still the main driving force behind the work being done.
Selenium IDE only works on Firefox because it is a Firefox add on.
I personally would avoid Rational Functional Tester because it has a lot of weaknesses that its not even worth contemplating.
If you start with Selenium there are some tutorials on my site at http://www.theautomatedtester.co.uk
Try Sahi (http://sahi.co.in/) It works across browsers and operating systems. It has a powerful recorder, and great APIs for object identification. It supports HTTPS, proxy tunneling etc. and has drivers in sahi script, java and ruby. It also has parallel playback inbuilt. It is 5 yr old mature project hosted on SourceForge, with releases almost every month.
It automatically waits for AJAX and page loads, and does not use XPaths for object identification. It also handles sites with dynamic ids.
Selenium is probably your best bet out of the tools you mentioned. What are the issues it has with IE8? You might want to check out HttpUnit to see if that meets your needs, also.
Selenium RC is a great tool if you invest the time to use it. With significant modifications to the existing library I've gotten it to fulfill all of my front end testing needs.
The confusion you are having about Webdriver is understandable. Selenium 2 is in development and will be a merge of Webdriver and Selenium. Check out: http://www.youtube.com/watch?v=RQD4EzWI4qk to get more detail.
The only browser that I have found to be unusable with Selenium is IE6. IE7 and IE8 work fine as does Firefox (which I have modified to include firebug for debugging purposes).
I'm in the same boat. It is a difficult problem to solve. Windmill and Selenium are the 2 best I've found. Though they both have issues. Selenium can only record scripts in Firefox and I haven't managed to get the proxy chaining to work as advertised. Windmill you can record in any browser and you can supposedly tweak the proxy to put extra logic in there, but the js mechanism for recording across page loads has been in my experience very brittle at least on the app I have to test.
I don't think anyone can get it quite right as long as there is more than one browser that needs to be supported.
Maybe have a look at SIKULI. It's a different paradigm but, depending on what you want to test exactly, it may do the job and will work with any browser, on any platform.
Have a look at their official blog for some examples of interactions with web applications.
So I wrote some of my more problematic tests in Selenium RC, using the Python driver. It was a better experience than writing the same tests in pure JavaScript, but I still had some of the same issues.
Testing something like an ajax autocomplete widget, meant forking some of the code depending on IE, or Firefox, and I still can't get typeKeys or a combination of type with typeKeys to work in Safari.
So, I am not sure if having cross browser clean, extensive ui tests is a bit unrealistic.
Should I try webdriver/Selenium 2? Would that make things better, or is that product not ready for prime time yet? How's the Python binding for that? I don't know Java, but I would learn some if need be.