What are the differences? What are the advantages of using one over the other for an Angular project?
Nightwatch.js vs Protractor
If you are working on an AngularJS project, the choice is simple - Protractor:
it is being made specifically for angular apps (though it can be used for non-angular apps also)
supports angular-specific locator strategies (like by.model, by.repeater etc)
waits for angular to start up during the page load (sync)
it is being actively maintained and improved mostly by google developers with a close cooperation with an angular team - this means that protractor follows the angular release cycle and new changes
There are a few more differences that weren't laid out in the top answer.
The big one for me is, as of now, protractor doesn't support phantomJS (http://www.protractortest.org/#/browser-setup#setting-up-phantomjs). I planned on integrating automation in to the CI system, but with protractor I would have to set up a windows VM to run the tests.
Other benefits of nightwatch:
Asserts are done automatically.
Clean console output, you could hand
off to anyone.
Test reports are automatically generated and saved to a /reports directory.
Protractor pros:
The angular-specific locators that protractor supports is very powerful and could help immensely with creating stable tests.
WaitForAngular is also very helpful.
EDIT:
Since the time I wrote this comment and present day our team has happily switched over to using cypress.io. If you are deciding test frameworks I highly recommend checking it out: https://www.cypress.io/
Related
I kind of think that answer of this question would be No by going through Protractor issue 1798 and Protractor vs Webdriver-IO comparisions.
I want to write automated tests cases for a hybrid mobile app in JavaScript which will run on Appium.
Currently I have both Protractor and Appium configured in same project to run those End-to-End test cases, and they work.
Problems are that,
I have to write separate test cases for each of them.
Cases should also work on mobile devices(Android and iOS) and protractor doesn't support native apps tests.
Situation is, I am more comfortable with Protractor's settings in my current project than Webdriver-IO which Appium uses (just a personal preference).
Question-
Is there a way of using only Protractor while writing test cases which would work using Appium and work perfectly on Devices/Emulators?
I am also open for any suggestion(s).
If answer remains No, I will change my codes to only use WebdriverIO in order to keep my test cases reusable and only once.
The answer is still NO protractor currently does not have support for mobile native apps but you can use it for automating browsers in your mobile.
Better use WebdriverIO but it also has its own limitations, please do check its changelog and github issues before deciding it as your webdriver framework.
I have just started to use protractor. I wanted to know it's limitation and specifically if it is dependent on version of AngularJS
I don't complete agree with #Zanon here. Protractor actually relies on some Angular-specific things like getTestability test helper that helps to let Protractor know that Angular is stable, ready and there are no outstanding requests left (source). This is actually what makes Protractor somewhat special and is recommended to use for testing Angular apps - the test flow becomes more natural, there is usually no need for extra waits - Protractor just works in sync with Angular application under test.
To answer your specific question:
for testing Angular 1.x apps - you can use any Protractor version
for testing Angular 2.x apps - Protractor >= 2.5.0
No, Protractor is not dependent on any specific Angular 1.x version.
However, since Angular 2.x added big new and breaking changes, you need Protractor version 2.5 (or later) to use it (you can confirm this here).
Protractor is a Node.js program that uses browser drives to execute tests as a real user would. It was designed thinking on Angular workflows, but it doesn't use Angular code, so its dependency is by design choices and not on specific code versions.
Here's the situation:
We're a Microsoft shop using Visual Studio / TFS etc.
We have a build server that nightly performs builds for us and runs various tests to ensure nothing has been broken during the day
We're now doing more and more JavaScript and this is likely to continue.
None of our JavaScript is currently tested because as I've looked around I've found the JavaScript testing world is in a state of flux. There are lots of offerings which I've read about and I'm not sure which path to take. I found this question quite helpful in terms of just raw info:
JavaScript unit test tools for TDD
Ideally I'd like to use something which integrates nicely into Visual Studio and our TFS build server. Most of the JavaScript testing options do not seem to support this.
I found this article: (but I'm not too keen on using jscript for testing)
http://stephenwalther.com/blog/archive/2010/12/20/integrating-javascript-unit-tests-with-visual-studio.aspx
And I found this - which looks quite promising:
http://chutzpah.codeplex.com/
So what I'm wondering is, is there any consensus yet? Are people adopting one tool over another, is any approach gaining critical mass yet? I'm keen to make use of something which is clearly working for a lot of people already.
Is it worthwhile me sticking it out for a JavaScript testing framework that plays nice with Visual Studio / TFS or should I look to use one of the non Microsoft friendly offerings instead?
All insights appreciated!
I wouldn't got that far and state that I'm moving away from developing in MS environment.
I think Visual Studio is a great tool. Working a little with Eclipse & XCode, for PhoneGap development even gave me another perspective as to how Visual Studio really rocks.
Regarding JavaScript, there are many enhancements in Visual Studio 2012, but integral solution for Unit Testing is not yet part of it.
After being involved in some JavaScript heavy projects, also felt the need for Unit Testing, so started with In-Browser tests with QUnit and Sinon.JS.
I went out there to find the right stack to meet my expectations:
Seamless integration with Visual Studio.
So we can run tests without leaving the IDE.
Headless testing.
Against multiple browsers at the same time if possible.
Support for AMD (Asynchronous Module Definition) / Require.JS.
Support for Async Tests.
Documented libraries & Tools.
Free Tools are preferred.
Eventually find my perfect combination for Headless testing from inside the IDE with JS-Test-Driver.
I've written a 3 parts series on my blog:
Part I - Expectations & the selected Stack
Part II - Setting up the environment for Headless tests
Part III - Testing AMD/Require.JS Modules
You can find the last part here, with links to first 2 posts:
http://www.upstruct.net/2013/02/javascript-unit-testing-in-visual-part-3.html
Been using this combination in several projects, and got a very good feedback from the teams so far.
Let me know if it suits you...
I've been looking around for the same answer but am now wondering if I should move away from developing in an MS environment altogether. As more and more of what I do is on the client side, Visual Studio seems less relevant. Funnily enough though, since I've been playing with Eclipse I've had the same problem - lack of JS testing plugins! So I'm not sure there's a consensus outside of MS development. I think part of this is a reluctance of seasoned developers of C# / Java to use JS but rather they want to use a strongly typed language which compiles (using tools like GWT and Script Sharp). The result is confusion about what comprises best practice. But since the tools for JS seem to be improving I'm somewhat reluctantly coming to the conclusion that JS should be the first point for coding rather than adding another layer of abstraction. Thus I'm resigned to using their available tools - QUnit for the moment is my choise.
In 2021 it seems that Microsoft is recommending we use a NodeJS app to run our JavaScript Unit Tests. Visual Studio's test window can now see tests we write in that project even when the overall project is in C#:
https://learn.microsoft.com/en-us/visualstudio/javascript/unit-testing-javascript-with-visual-studio?view=vs-2019&tabs=mocha
One of the testing library combos (unit testing / mocking / assertions) recommended is using Mocha / Sinon / Chai. For this flavor there's a useful tutorial located here:
https://scotch.io/tutorials/how-to-test-nodejs-apps-using-mocha-chai-and-sinonjs
I've never used Selenium but I guess it's for simulating user interaction in all browsers.
That's like integration tests.
But how do you test your js libraries/frameworks (unit testing) on all the browsers in an automated way?
For unit-testing you can try http://code.google.com/p/js-test-driver/
JsTestDriver consist of a single JAR file which contains everything you need to get started. For in depth discussion of command line option see GettingStarted.
Here is an overview of how JsTestDriver works at runtime...
You can have a look at TestSwarm:
project that I’m working on: TestSwarm...
Its construction is very simple. It’s a dumb JavaScript client that continually pings a central server looking for more tests to run. The server collects test suites and sends them out to the respective clients.
All the test suites are collected. For example, 1 “commit” can have 10 test suites associated with it (and be distributed to a selection of browsers)...
The nice thing about this construction is that it’s able to work in a fault-tolerant manner. Clients can come-and-go. At any given time there might be no Firefox 2s connected, at another time there could be thirty. The jobs are queued and divvied out as the load requires it. Additionally, the client is simple enough to be able to run on mobile devices (while being completely test framework agnostic)...
The best one imo is the one from YUI : http://developer.yahoo.com/yui/3/test/
But doing unit-testing in every browser is kind of hard... Most people just test with it during development and just use node.js to test later on in case they broke something.
As the referenced post in a previous article suggests, you could use js-test-driver.
Its specifically for JavaScript unit testing across multiple browsers, exactly what you want. I have messed around with it and it is pretty good. I haven't done any serious commercial testing in it though.
For unit testing you should look at solutions that do not load up a browser to do the tests.
You can look at something like RhinoUnit for that - http://code.google.com/p/rhinounit
Also have a look at Dojo Object Harness (DOH) unit test framework http://dojotoolkit.org/reference-guide/util/doh.html
Look at a similar question here which can give you an idea on how to TDD js - JavaScript unit test tools for TDD
It's no longer actively maintained, but I've still been happy with JSUnit for Javascript unit testing: https://github.com/pivotal/jsunit
It includes both an HTML/Javascript framework you can run in the browser, and a java-based test runner that you can invoke from ant.
To test multiple browsers in parallel you would Selenium Grid. Please take a look here: http://selenium-grid.seleniumhq.org/step_by_step_installation_instructions_for_windows.html for step by step instructions on how to use it.
I haven't used it so far, and it is still in beta, but FuncUnit declares itself as "A functional test suite based of qUnit, Selenium and jQuery".
There is also an infographic explaining how it works
Maybe it is something what you want? The github repo seems to be quite active.
You should check http://saucelabs.com its a cloud base selenium environment that allow you to build your test and then upload them, run them in as many browsers as you want.
You should consider the capabilities offered by crossbrowsertesting.com
(It is not for free). For js testing try JsUnit (http://www.jsunit.net/).
Quoted from its homepage
JsUnit is a Unit Testing framework for client-side (in-browser) JavaScript. It is essentially a port of JUnit to JavaScript. Also included is a platform for automating the execution of tests on multiple browsers and mutiple machines running different OSs.
As people have already said so, you should use JsTestDriver. It has it's own test system, but you can use other test libraries with it, for example Jasmine ( http://pivotal.github.com/jasmine/ ). You can find a list of adaptors for JsTestDriver here: http://code.google.com/p/js-test-driver/wiki/XUnitCompatibility
I'd go for QUnit, which is what jQuery uses. I've ran it on lots of desktop browsers as well as iPhones and Android phones.
There's some great tutorials and it can be integrated easily with things like js-test-driver. QUnit is modeled after JUnit and all the other xUnit testing frameworks (like PHPUnit) so it's easy to pick up the API.
The basic syntax is as follows:
test("my first test", function() {
var str = "hello";
equals( "hello", str, "Should be hello" );
});
It also looks quite nice:
There are few companies that specialize in cross browser testing.
http://browserling.com/
http://www.browserstack.com/
http://saucelabs.com/ (already mentioned here)
http://browsershots.org/
http://www.browsercam.com/
http://browserseal.com/
Use whatever testing lib you want.
Selenium or SauceLabs etc are not unit testing. They are functional/integration testing solutions.
You need to abstract your external usages like DOM to unit test javascript.
Write your tests so that they can use any external library like jquery by configuration. So that, you can unit test your logic without touching any externality and you can also both test cross-browser testing.
We have a web application that makes extensive use of AJAXy Javascript in the UI. We have nearly complete code coverage of our backend using Shoulda and Webrat, and would like to extend our test suite to include full integration testing through the Javascript UI.
We tried Selenium but found it brittle and temperamental. Are there more reliable options?
UPDATE
For those still checking this out, we ended up using Xvfb so we can run Firefox without a screen. Allows us to run the test on a headless Jenkins CI server. We still have to run tests "live" locally occasionally to debug, but it works pretty well.
One of the JavaScript gurus where I work recently pointed out PhantomJS as an interesting tool for testing our JavaScript-heavy web applications. We haven't tried it out yet but the idea of a headless WebKit for DOM testing sounds promising to me.
This is something I have been wrestling with for a while, as I am doing some work with ExtJS (a very powerful JavaScript UI builder for the browser) and Rails.
After having researched quite a few different options. I still haven't found a perfect solution for it. Ideally, I would be able to run them headless and just report on the output. Unfortunately, none of the emulators out there seem to be able to run JavaScript with full DOM support seamlessly (at least, none of the options I've found are). So that pretty much means that you have to run your full-powered JavaScript code in a real interpreter (such as a browser). Webrat with Selenium works acceptably well, assuming you're willing to deal with the pain of trying to path out your requests to the UI properly. If it's your own JavaScript that you're implementing it against, that may be easier. But when it comes to a third party UI library that you don't have much control over, it can certainly get, shall we say, interesting.
Probably not the most helpful response, but that has been my findings up to now!
Hmm I would give Capybara a look, it can use selenium-webdriver (not to be confused with selenium-RC, they are different) for javascript testing. I haven't found it very brittle when compared with Webrat... it seems to be fairly consistent.
As Chris Rueber says, there aren't really any headless DOM interpreters that support JS as well - for now it's fire up a web browser for your automation or write unit tests in the javascript itself (Which isn't really integration testing either).
When you have a lot of selenium-webdriver-backed tests they can take awhile to run sometimes, but it's surely better than no tests at all.
check out the gem jasminerice to test your js logic.
https://github.com/bradphelan/jasminerice
for the integration test I would recommend to use rspec with capybara as acceptance tests. distinguish request specs and acceptance specs!
another possibility is to use turnip as an alternative to cucumber.
https://github.com/jnicklas/turnip
to speed up your tests test headless. You could use capybara-webkit (depends on qt) or poltergeist (which depends on phantomjs).
both are easily to set up. I prefer poltergeist.
There are a couple of gems you could use if you didn't like Selenium.
The one I recommend is Jasmine: https://github.com/pivotal/jasmine
You can also check out Culerity: https://github.com/langalex/culerity