I'm using Protractor (v 1.3.1) to run E2E tests for my Angular 1.2.26 application.
But sometimes, tests are ok, sometimes not. It seems that sometimes the check is done before display is updated (or something like "synchronisation" problem).
I try many options :
add browser.driver.sleep instructions,
disable effects with browser.executeScript('$.fx.off = true')
add browser.waitForAngular() instructions
without success.
What are the bests practice to have reliables E2E tests with protractor?
JM.
Every time I have similar issues, I'm using browser.wait() with "Expected Conditions" (introduced in protractor 1.7). There is a set of built-in expected conditions that is usually enough, but you can easily define your custom expected conditions.
For example, waiting for an element to become visible:
var EC = protractor.ExpectedConditions;
var e = element(by.id('xyz'));
browser.wait(EC.visibilityOf(e), 10000);
expect(e.isDisplayed()).toBeTruthy();
Few notes:
you can specify a custom error message in case the conditions would not be met and a timeout error would be thrown, see Custom message on wait timeout error:
browser.wait(EC.visibilityOf(e), 10000, "Element 'xyz' has not become visible");
you can set EC to be a globally available variable pointing to protractor.ExpectedConditions. Add this line to the onPrepare() in your config:
onPrepare: function () {
global.EC = protractor.ExpectedConditions;
}
as an example of a custom expected condition, see this answer
Another point which is very important in testing with Protractor is understanding the ControlFlow. You may find explaination and code example here : When should we use .then with Protractor Promise?
Jean-marc
There are two things to consider.
The first is that you should properly sequence all protractor actions (as also hinted by #jmcollin92). For this, I typically use .then on every step.
The second important thing is to make sure that a new test it(...) only starts after the previous it(...) has completed.
If you use the latest version of Protractor, you can use Jasmine 2.x and its support for signalling the completion of a test:
it('should do something', function(done) {
clicksomething().then(function() {
expect(...);
done();
});
});
Here the done argument is invoked to signal that the test is ready. Without this, Protractor will schedule the clicksomething command, and then immediately move on with the next test, returning to the present test only once clicksomething has completed.
Since typically both tests inspect and possibly modify the same browser/page, your tests become unpredictable if you let them happen concurrently (one test clicks to the next page, while another is still inspecting the previous page).
If you use an earlier version of Protractor (1.3 as you indicate), the Jasmine 1.3 runs and waitsFor functions can be used to simulate this behavior.
Note that the whole point of using Protractor is that Protractor is supposed to know when Angular is finished. So in principle, there should be no need to ever call waitForAngular (my own test suite with dozens of scenarios does not include a single wait/waitForAngular). The better your application-under-test adheres to Angular's design principles, the fewer WaitForAngular's you should need.
I would add that disabling ngAnimate may not be enough. You may also have to disable all transition animation by injecting CSS (What is the cleanest way to disable CSS transition effects temporarily?).
Related
Jest uses describe and it and expect without you having to require them. This is okay because if you have a test file called test.spec.js, you'll never execute it directly by issuing the command node test.spec.js.
I want to execute it using node as a standard js file, without having to use jest's cli or npm test. Is it possible to do that?
For instance, I'd convert the following file:
// taken from documentation
const user = require('./users.js')
it('works with promises', () => {
expect.assertions(1);
return user.getUserName(4).then(data => expect(data).toEqual('Mark'));
});
To something like
var {describe, it, expect} = require('jest-primitives')
const user = require('./users.js')
it('works with promises', () => {
expect.assertions(1);
return user.getUserName(4).then(data => expect(data).toEqual('Mark'));
});
So that it's a self-contained js file, which can be run using just node.
A complete list of globals that jest provides is given here.
Is it technically possible? Yes.
Should you? Probably not. (But there are better ways - tl;dr use tape)
Background
Jest is a test harness.
Jest is not the only test harness to use describe() and it(). These are typical test keywords for behavior-driven development (BDD). You'll also find them used with Mocha and others.
describe() and it() are functions that interface with the test harness, telling it to add a test suite and a test case, respectively. The test harness then runs the test cases, collects the results, formats the results and outputs them.
Why you shouldn't do this with Jest
Generally, you should use technology as idiomatically as possible. This makes it easier for others to read, understand, and work with your technology.
In particular, using Jest in this way will be self-implemented, hacky, buggy and generally incomprehensible to anyone not familiar with your code. That being said, it should be possible.
How you might attempt that with Jest
Jest defines runner packages in their Jest monorepo. One is Circus, the other is Jasmine2.
Circus exports describe(), it() and some other keywords, though these are not really useful to us, as these functions only internally create the test suites and test cases but do not expose them to us or give us a way to run them.
Jasmine2 exports an executable function which returns a Promise of a test result. The code for jasmineAsyncInstall creates most of the keywords in the env or globally, and you might be able to use these.
What you'd want to do here is define it() and describe() functiions, either globally, as exports (if you'd like to use them as in the code sample in the question), or hackily by defining them inside the scope of the main module. These functions should register test cases and test suites. You'll want to either keep track of the test cases and run them later, or run them right away and keep track of the test results.
The problem now lies in determining when the test module has finished running. That is, when all of the describe() and it() have executed (whether or not the test cases themselves have executed), as well as any other incident code (code that isn't in any block). There is no good way to handle this, and here's where it may get hacky again. The easiest way is probably to add a listener to process.on('exit'.
Why that doesn't matter
A test harness is generally just a test runner and a reporter. Jest, in particular, is just a collection of components, all of which are configurable.
If we're just pulling a function here and a variable there from Jest, can we really say that we're still using it? Why do we even want to? There's really no reason to use Jest here. If you don't like the way it runs tests, you should use a different test harness instead of trying to modify it. If you like the reporter, Jest exports a package containing only the reporter.
A better way to make test files runnable
Use tap or tape. These are designed to be run the way you want, and are configurable.
Example:
const test = require('tape');
const MyClass = require('../src/my-class');
test('MyClass.doSometing should be true', (t) => {
const result = MyClass.doSomething();
if (result === true) {
t.pass('The test passed! Hooray! Our class MyClass is seemingly error-free!');
} else {
t.fail('Oh noes. Our test has failed. Why am I such a bad programmer....?');
}
t.end();
});
I had to do a lot of research to get to the right library.
The most suitable one that I found so far is Jest Lite. This guy has done a great job.
Another solution I found is a workaround with Chai.js. Install chai and import the expect from the library. Run expect statement. Unfortunately, it does not return a promise or result. You can just get the job done with a try-catch.
I am trying to figure out how to restrict my tests, so that the coverage reporter only considers a function covered when a test was written specifically for that function.
The following example from the PHPUnit doc shows pretty good what I try to achieve:
The #covers annotation can be used in the test code to specify which
method(s) a test method wants to test:
/**
* #covers BankAccount::getBalance
*/
public function testBalanceIsInitiallyZero()
{
$this->assertEquals(0, $this->ba->getBalance());
}
If the test above would be executed, only the function getBalance will be marked as covered, and none other.
Now some actual code sample from my JavaScript tests. This test shows the unwanted behaviour that I try to get rid of:
it('Test get date range', function()
{
expect(dateService.getDateRange('2001-01-01', '2001-01-07')).toEqual(7);
});
This test will mark the function getDateRange as covered, but also any other function that is called from inside getDateRange. Because of this quirk the actual code coverage for my project is probably a lot lower than the reported code coverage.
How can I stop this behaviour? Is there a way to make Karma/Jasmine/Istanbul behave the way I want it, or do I need to switch to another framework for JavaScript testing?
I don't see any particular reason for what you're asking. I'd say if your test causes a nested function to be called, then the function is covered too. You are indeed indirectly testing that piece of code, so why shouldn't that be included in the code coverage metrics? If the inner function contains a bug, your test could catch it even if it's not testing that directly.
You can annotate your code with special comments to tell Istanbul to ignore certain paths:
https://github.com/gotwarlost/istanbul/blob/master/ignoring-code-for-coverage.md
but that's more for the opposite I think, not to decrease coverage if you know you don't want a particular execution path to be covered, maybe because it would be too hard to write a test case for it.
Also, if you care about your "low level" functions tested in isolation, then make sure your code is structured in a modular way so that you can test those by themselves first. You can also set up different test run configurations, so you can have a suite that tests only the basic logic and reports the coverage for that.
As suggested in the comments, mocking and dependency injections can help to make your tests more focused, but you basically always want to have some high level tests where you check the integrations of these parts together. If you mock everything then you never test the actual pieces working together.
Recently, I've noticed that one of our tests has the following line:
browser.actions().sendKeys(protractor.Key.RETURN);
The intent is understandable, but it would actually do nothing since perform() was not called. For some reason, the test itself was passing which indicates a problem in the logic of the test and the following expectations.
How can I spot this kind of problems as early as possible and, ideally, prevent this protractor/WebDriverJS usage violation to be committed into the repository?
One option would be to use static code analysis - there is an ESLint linting utility that has a set of different plugins. Nowadays, there is a eslint-plugin-protractor plugin that, aside from other protractor-specific violations, would catch browser.actions() without perform().
Here is the output of a ESLint run in this case:
/Users/user/job/app/specs/test.spec.js
36:13 error No perform() called on browser.actions() protractor/missing-perform
I expect this could be achieved via a (scriptable) editor plugin or linter rule.
That said, surely the best way to evaluate the test script is to run it for real - but also to ensure that all test actions have a corresponding assertion / validation.
Your Key.RETURN must presumably have some effect on the DOM, or initiate some action, the result of which can be detected (page changes, data changes, etc.) and which is probably meaningful and easier to read than a static analysis rule.
I’m trying to run my angular e2e tests with protractor. I’ve got some situations where a select exists inside a modal. Depending on the machine running the tests, this sometimes fails as protractor can’t find the select with:
NoSuchElementError: No element found using locator: By.cssSelector("div#s2id_items”)
On a slower machine this works every time, while on faster machines it often fails. My guess is that the modal is still being animated when protractor tries to access the selector, hence, resulting in failure.
I’ve tried to disable animations without success with the code bellow inside the onPrepare directive in my protractor config:
var disableNgAnimate = function() {
angular.module('disableNgAnimate', []).run(['$animate', function($animate) {
$animate.enabled(false);
}]);
};
browser.addMockModule('disableNgAnimate',disableNgAnimate);
I’m using angular 1.4.3 with bootstrap 3.3.5 and protractor 2.1.0.
Thanks
Edit:
1 - I'm not using explicit waits and I wouldn't like to, as these would either considerably slow down tests or still be prone to failure in some scenarios.
You could try using Expected Conditions for wait, such as:
var EC = protractor.ExpectedConditions;
var myElement= element(by.css('div#s2id_items'));
browser.wait(EC.presenceOf(myElement), 5000);
//rest of your code
This wait will not slow down your tests, as it will only wait long enough for the element to be shown, and fail after 5 sec if it isn't.
EDIT: For clickable, animated objects you can either try built-in "elementToBeClickable" condition (just replace presenceOf in the above example), or write your own, that would do whatever you like (function returning true of false). E2E tests should "think" as a user would, and a user would wait for the animation to end, so maybe it would be best if you used explicit wait for the animation after all.
I am interested in how to globally mock out the file picker in the browser. Specifically, I am most interested in doing this in Firefox but would prefer a general solution.
I only care about preventing the file picker dialog from appearing. I do not need to be able to assert that it did open. The problem is that I have unit tests for JavaScript code that open the file picker. When the dialog opens, it halts the execution of the test suite.
An example situation is that I am testing the onRender method of a Backbone.View. That method renders a subview, which will open the File Picker when it is rendered. Since I am not directly testing that subview, I would prefer not to mock out portions of its behavior when I am only interested in unit testing some other part of the onRender method.
Example:
//Test file
it("should do something", function() {
var view = new App.Views.SomeView();
spyOn(view.modelBinder, "bind");
view.render();
expect(view.modelBinder.bind).toHaveBeenCalled();
});
//View file
onRender : function () {
this.modelBinder.bind(this.el, this.model);
this.$("#thing").html(this.subview.render().el); //This line has a side effect that opens file picker
}
Essentially, I do not want to explicitly mock out the behavior that causes the file picker to be opened because it is not what I am interested in testing here. Doing so will make the test suite much more brittle and difficult to maintain.
Use sinon to mock/spy/stub out the calls. You can test for the calls being made instead of actually making the calls.
That way you can test that the function has been called without calling the actual function that displays the dialog.
To answer your question: Just don't.
I would replace subview.render() with an empty function to avoid the undesired side effect. You say however:
"I do not want to explicitly mock out the behavior that causes the
file picker to be opened because it is not what I am interested in testing..."
Which is a little contradictory. If you want to Unit-Test App.Views.SomeView, you will have to mock out external collaborators, specially when not interesting, and including your file picker. On the other hand, you should not mess with the SUT when unit-testing it.
Mocking would in fact make your test more prone to be red, but is the only way to make sure your production code does not suffer from ill forms of coupling (IMHO, A common pitfall with Backbone.js apps.)
The only place in which you would need to avoid the file-picker to display, is when unit-testing your file-picker itself, in that case you could use sinon as sugested or leave it without coverage if you are using jQuery. Remember the "Never mock a type you don't own" rule.