I am working with Protractor 5.4.0 and cucumber.
The protractor.conf.js file is:
global.expect = require('chai').expect;
var chai = require('chai');
var chaiAsPromised = require('chai-as-promised');
chai.use(chaiAsPromised);
exports.config = {
seleniumAddress: 'http://localhost:4444/wd/hub', // This is targetting your local running instance of the selenium webdriver
specs: [
'../Features/UI_Tests.feature'
],
capabilities: {
browserName: 'chrome' // You can use any browser you want. On a CI environment you're going to want to use PhantomJS
},
framework: 'custom', //We need this line to use the cucumber framework
frameworkPath: require.resolve('protractor-cucumber-framework'), // Here it is
cucumberOpts: {
//format: 'pretty',
require: '../Features/step_definitions/my_steps.js', // This is where we'll be writing our actual tests
// tags: ['#basic'],
strict: true,
plugin:"json"
},
resultJsonOutputFile:'./testResults.json', //output file path to store the final results in .json format
params: {
env: {
hostname: 'http://0.0.0.0:8000' // Whatever the address of your app is
}
}
};
I have this scenario defined with some examples:
Scenario Outline: dropdown boxes appear and work as expected.
When go to "URL"
Then application is running
When click <box>
Then <option> is present in <box>
Examples:
|box| option|
|templateSelection| Apparent Energy |
|templateDeliveryPathSelection| Email |
|templateLocaleSelection| English |
I am using this piece of code to check that text of the dropdown box is the same one as the option column:
checkDropdown: function (value,dropdown) {
var text = element(by.id(dropdown)).getText();
expect(text).to.eventually.equal(value);
},
It seems to be working properly because the output informs that all the scenarios have passed. However, if we change any of the values in the "option" column to make it fails, the output is the same one, all scenarios pass. Why?
Thanks in advance.
Try:
checkDropdown: function (value,dropdown) {
var text = element(by.id(dropdown)).getText();
return expect(text).to.eventually.equal(value);
}
It's because you aren't returning the expect. You always have to return a promise in your step definitions. if the function returns undefined it will pass regardless.
I highly recommend using https://github.com/canvaspixels/cucumber-protractor if you're using those two technologies to save tons of boilerplate and effort. You can keep your existing tests and migrate gradually.
There's a demo video here: https://www.youtube.com/watch?v=5vuYL4nxMXs
Related
I am running Karma/Jasmine/Angular 2.0 tests on my development box. Just recently, Jasmine on my development box decided to start running my tests three times. Yes, exactly three times, every time.
On the first run, everything passes as expected. However, on the second and third pass, all of the same things fail. It always acknowledges that there are 7 tests, but runs 21, and 10 fails (first-grade math out the window)????
This also fails on Travis with SauceLabs. (Note: That links to an older build with 3 tests, but ran 9, and 5 fail???)
I have a screenshot, karma.conf.js file, and one suite which started this whole thing. Any help with be greatly appreciated.
Culprit [TypeScript] (Remove this and problem solved on my dev box):
Full source
describe('From the Conductor Service', () => {
let arr: Array<ComponentStatusModel> = null;
let svc: ConductorService = null;
beforeEach(() => {
arr = [/* Inits the array*/];
svc = new ConductorService();
});
describe('when it is handed a container to hold objects which need to be loaded', () => {
// More passing tests...
/// vvvvv The culprit !!!!!
describe('then when you need to access the container', () => {
beforeEach(() => {
svc.loadedContainer = arr;
});
it('it should always be available', () => {
assertIsLocalDataInTheService(arr, svc.loadedContainer);
});
});
/// ^^^^^ End of culprit !!!!!
});
// More passing tests...
});
Failing Tests:
Browser Screenshots:
Not sure if this is related, but before all of the errors happen, the Jasmine call stack is smaller (left, observe scrollbar). After the errors start, the stack just gets bigger with repeating calls to the same functions (right, observe scrollbar).
Suite Stack is Wrong:
In my test, the Nanobar and Conductor spec files are totally separate. However, you can see the suites array includes stuff from the Nanobar and Conductor specs. Somehow Jasmine mashed these two spec files together (after everything started failing), and resulted in my describe() statements not making any sense when published to the console.
Simplified karma.conf.js:
Full source
module.exports = function (config) {
config.set({
autoWatch: false,
basePath: '.',
browsers: ['Chrome'],
colors: true,
frameworks: ['jasmine'],
logLevel: config.LOG_INFO,
port: 9876,
reporters: ['coverage', 'progress'],
singleRun: true,
coverageReporter: {
// Code coverage config
},
files: [
// Loads everything I need to work
],
plugins: [
'karma-chrome-launcher',
'karma-coverage',
'karma-jasmine'
],
preprocessors: {
'app/**/*.js': ['coverage']
},
proxies: {
// Adjust the paths
}
})
}
Can you try refreshing your browser in your first assertion in each of your test files?
Try this:
browser.restart();
I had the same problem and this fixed it for me.
The first thing is these test run randomly. If you pass some data in any test case if you think you can resue that it is not possible.
You have to declare the data in before each so all test cases get data.
All test cases run independently.
If you are using array or object you must have to use this after deep cloning because array and object works on the reference. If you manipulate any value it will also change the original array.
In most of the cases if the test fails there may be an error of data you are passing in test cases.
I would try to debug this and pinpoint the exact cause.
Usually happens when I have redirection code or any reload code inside the functions I'm testing.
You can try adding an f to the prefix of describe and it (i.e. fdescribe and fit)
Since I'm a newbie with automated tests and protractor, I'm having some trouble setting this up in my tests.
According to the guide, every time that I create a new instance of screenshot reporter, I have to pass a directory path. Right, this means that every time I create a new instance in my spec file?
Also, there are functions to take screenshots of my skipped and my failed tests. Where i supposed to use takeScreenShotsForSkippedSpecs and takeScreenShotsOnlyForFailedSpecs? In my config file?
This is my onPrepare:
onPrepare: function () {
browser.driver.manage().window().maximize();
global.dvr = browser.driver;
global.isAngularSite = function (flag) {
browser.ignoreSynchronization = !flag;
}
jasmine.getEnv().addReporter(new ScreenShotReporter({
baseDirectory: '/tmp/screenshots',
takeScreenShotsForSkippedSpecs: true,
takeScreenShotsOnlyForFailedSpecs: true
}));
Note: If you are using jasmine2, use protractor-jasmine2-screenshot-reporter.
For jasmine1:
I've been using successfully using protractor-html-screenshot-reporterpackage. It is based on protractor-screenshot-reporter, but also provides a nice HTML report.
Here is what I have in the protractor config:
var HtmlReporter = require("protractor-html-screenshot-reporter");
exports.config = {
...
onPrepare: function () {
// screenshot reporter
jasmine.getEnv().addReporter(new HtmlReporter({
baseDirectory: "test-results/screenshots"
}));
},
...
}
After running tests, you would get an HTML file containing (example):
You can click "view" to see the test-case specific screenshot in the browser.
The readme in the library is pretty self explanatory. After installing the library, add it onto protractor's onPrepare in your protractor config file.
i.e.
protractorConf.js:
var ScreenShotReporter = require('protractor-screenshot-reporter');
exports.config = {
// your config here ...
onPrepare: function() {
// Add a screenshot reporter and store screenshots to `/tmp/screnshots`:
jasmine.getEnv().addReporter(new ScreenShotReporter({
baseDirectory: '/tmp/screenshots',
takeScreenShotsForSkippedSpecs: true
}));
}
}
then protractor protractorConf.js to run protractor.
Just recently I published a brand new plugin called protractor-screenshoter-plugin that captures for each browser instance a screenshot and console logs. The snapshot is made optionally for each expect or spec. It comes with a beautiful angular and bootstrap based analytics tool to visually check and fix tests results.
Check it out at https://github.com/azachar/protractor-screenshoter-plugin.
Also, I created a list of all available alternatives, so if you find something else, please do not hesitate to add it there.
This is a follow-up to the Set firefox profile with protractor topic.
According to the setFirefoxProfile howto, it is possible to set a firefox profile with a special "helper" js code which uses firefox-profile and q libraries to make an encoded firefox profile on the fly.
This worked for me until I've tried to use multiple browsers and configuring multiCapabilities:
exports.config = {
seleniumAddress: 'http://localhost:4444/wd/hub',
multiCapabilities: [
{
browserName: 'chrome',
specs: [
'footer.disabledCookies.spec.js'
],
chromeOptions: {
prefs: {
'profile.default_content_settings.cookies': 2
}
}
},
...
// other capabilities here
...
helper.getFirefoxProfile()
},
...
}
With this setup I'm getting an error (full traceback here):
Spec patterns did not match any files.
As I understand, this means that the setup with firefox profile is missing specs key. In other words, it cannot find any tests to run.
I've tried to include specs into the capabilities dictionary inside the helper itself, but the error persists.
How to fix the error and set firefox profile if using multiCapabilities?
As a workaround, I've created a separate protractor configuration file with only firefox configured (using capabilities) and set grunt to run protractor twice - one for this "firefox with a profile" config and the other one for all other browsers.
Right now, protractor can only accept promise as capabilities if we are NOT using multicapabilities. The reason for this is because multiCapabilities runs each task in a new process, so the promise (function) cannot be passed (single capabilities work because we're not forking).
Alternatively we could resolve capabilities in the launcher, before passing the resolved capabilities into the new processes; however, this will break the ability to set up proxies (https://github.com/angular/protractor/pull/1040), which relies on capability promises to be resolved after driverProvider setup.
I can't think of an easy way of doing this (without large refactoring), but it is definitely doable.
I created an issue for Protractor (https://github.com/angular/protractor/issues/1594). Please follow that and/or comment on it if this is something you need or you have other ideas to implement it.
For now you would need to use the workaround you mentioned in your original question.
UPDATE
https://github.com/angular/protractor/pull/1629 supports this. Starting in protractor 1.6 (or if you sync to master) you can pass in a function to config.getMultiCapabilities like onPrepare and onCleanup. This function can return a promise to multiCapabilties (i.e. array of capabilities).
See https://github.com/angular/protractor/blob/master/spec/getCapabilitiesConf.js for an example.
Following the pull request sent by #hankduan, here is how have I used getMultiCapabilities() to combine different capabilities where one of them is a promise (needed for firefox-profile to be set):
"use strict";
var FirefoxProfile = require("firefox-profile");
var q = require("q");
exports.config = {
seleniumAddress: "http://127.0.0.1:4444/wd/hub",
getMultiCapabilities: function() {
var deferred = q.defer();
var multiCapabilities = [
{
browserName: "chrome",
specs: [
"footer.disabledCookies.spec.js"
],
chromeOptions: {
prefs: {
"profile.default_content_settings.cookies": 2
}
}
},
{
browserName: "chrome",
specs: [
"*.spec.js"
],
exclude: [
"footer.disabledCookies.spec.js",
"footer.disabledJavascript.spec.js",
"footer.disabledFlash.spec.js"
]
},
{
browserName: "chrome",
specs: [
"footer.disabledFlash.spec.js"
],
chromeOptions: {
args: [
"--disable-internal-flash",
"--disable-bundled-ppapi-flash",
"--disable-plugins-discovery"
]
}
}
];
// Wait for a server to be ready or get capabilities asynchronously.
setTimeout(function() {
var firefoxProfile = new FirefoxProfile();
firefoxProfile.setPreference("javascript.enabled", false);
firefoxProfile.encoded(function (encodedProfile) {
var capabilities = {
"browserName": "firefox",
"firefox_profile": encodedProfile,
"specs": [
"footer.disabledJavascript.spec.js"
]
};
multiCapabilities.push(capabilities);
deferred.resolve(multiCapabilities);
});
}, 1000);
return deferred.promise;
},
...
};
Hope this would help somebody in the future.
As indicated in this stackoverflow answer, it looks like Karma will serve JSON fixtures. However, I've spent too many hours trying to get it to work in my environment. Reason: I'm doing angular testing and need to load mock HTTP results into the test, as Jasmine doesn't support any global setup/teardown with mock servers and stuff.
In my karma config file, I'm defining a fixture as so:
files: [
// angular
'angular/angular.min.js',
'angular/angular-route.js',
'angular/mock/angular-mocks.js',
// jasmine jquery helper
'jquery-1.10.2.min.js',
'angular/jasmine-jquery.js',
// our app
'../public/js/FooApp.js',
// our tests
'angular/*-spec.js',
// fixtures
{ pattern: 'node/mock/factoryResults.json',
watched: 'true',
served: 'true',
included: 'false' }
]
Before I even attempt to use jasmine-jquery.js in my jasmine test to load the JSON, I see karma choking on trying to serve it:
...
DEBUG [web-server]: serving: /Users/XXX/FooApp/spec/node/mock/factoryResults.json
Firefox 25.0.0 (Mac OS X 10.8) ERROR
SyntaxError: missing ; before statement
at /Users/XXX/FooApp/spec/node/mock/factoryResults.json:1
...
Here's what factoryResults.json looks like:
{ "why": "WHY" }
Any idea what's going on here? I see plenty of examples on the web of folks successfully loading JSON into jasmine tests via karma fixtures. Karma can see the file; if I put the wrong path in my fixture block, I see an error stating that it couldn't find any files that match my fixture pattern. I've tried reformatting the .json file in different ways... Any ideas?
Your problem is that 'false' has to be a boolean, not a string.
There is already an issue to validate the config better and fix such a mistakes.
Also, you might write a simple "json" preprocessor (similar to karma-html2js) that would make it valid JS and put the JSON into some global namespace so that you can keep the tests synchronous...
I also needed json fixtures in my karma test suite.
I ended up just using the html2js preprocessor with json files as well as html.
karma.conf.js:
module.exports = function (config) {
config.set({
frameworks: ["jasmine"],
files: [
'**/*.js',
'**/*.html',
'**/*.json',
'**/*.spec.js'
],
plugins: [
'karma-html2js-preprocessor'
]
preprocessors: {
'**/*.html': ['html2js'],
'**/*.json': ['html2js']
}
});
};
Then it is just a matter of getting the json from the __html__ global.
e.g.
var exampleJson = __html__['example.json'];
var jsonObj = JSON.parse(exampleJson);
var exampleHtml = __html__['example.html'];
document.body.innerHTML = exampleHtml;
So, I had a lot of issues with jasmine-jquery and I got a pretty decent workaround.
It's a little hacky, but it works. Basically, I just create a function accessible on the window, then stack the JSON fixtures inside a little switch:
if (typeof(window.fixtures === "undefined")) {
window.fixtures = {};
}
window.setFixture = function(type) {
var json;
if (type == "catalog") {
json = { ... }
}
if (typeof(type) !== "undefined") {
window.fixtures[type] = json;
}
return json;
}
Then, I can just stub it inline in the view:
describe "App.Models.Catalog", ->
it "provides the 'App.Models.Catalog' function", ->
expect(App.Models.Catalog).toEqual(jasmine.any(Function))
it "sets up a fixture", ->
setFixture("catalog")
console.log(fixtures["catalog"])
expect(fixtures["catalog"]).toBeDefined()
Boom, tests pass, and the object comes out in the log:
{
catalog_id: '2212',
merchant_id: '114',
legacy_catalog_id: '2340',
name: 'Sample Catalog',
status: '1',
description: 'Catalog Description ',
}
Now, it's accessible within my test.
It's of course not perfect or ideal, but I kept hitting strange matchErrors and the like with the jasmine-jquery plugin, and it's simple enough (and fast) for me to paste in a couple of JSON blocks and get moving.
You also save yourself the time fiddling around with the configuration and making any changes to the files for Karma.
Anyone have any better suggestions or have any luck getting jasmine-jquery to work?
I am using Karma to test my project and can see tests passing an failing in the console window, however, how do I get these to show in the browser? The browser only has a green bar (even though a test is failing) with
Karma v0.10.2 - connected
Written in it.
I have tried addong singleRun :false to the karma.config.js file.
The config file looks like this:
module.exports = function (config) {
config.set({
basePath: '../',
files: [
'app/lib/angular/angular.js',
'app/lib/angular/angular-*.js',
'test/lib/angular/angular-mocks.js',
'app/js/**/*.js',
'test/unit/**/*.js'
],
autoWatch: true,
singleRun: false,
frameworks: ['jasmine'],
browsers: ['Chrome'],
plugins: [
'karma-junit-reporter',
'karma-chrome-launcher',
'karma-firefox-launcher',
'karma-jasmine'
],
junitReporter: {
outputFile: 'test_out/unit.xml',
suite: 'unit'
}
})
}
matthias-schuetz wrote a plugin that claims to produce html test output.
https://npmjs.org/package/karma-htmlfile-reporter
Along with the instructions on the plugin page I had to include a reference to the plugin in the Karma config -
plugins: [
'karma-htmlfile-reporter'
]
Accroding to Documention, even not very perfect:
If 'singleRun' is false, it will set the ci-mode on, so, make it true, and you will see some failed red status on the top bars of browers.
There is no straight forward way to do this with Karma.
The "best" way to solve this problem would be to hunker down and write a html-reporter for karma (which would make a lot of us other Karma users very happy).
If this is too much work for you, the second best thing is to use the junit reporter which generates an xml file. You can then post-process the xml file in some way that turns it into a HTML-file which you can then view in your browser
I wanted to display HTML5 Web Notifications with Karma so I wrote something quick to get it to work with Karma version 0.11. Might behave slightly different with other versions. I load this script in with the rest of my application scripts, it will store the karma test results and after completion it will determine the success of the test and then reset to the original karma functions so they're not changed when this script gets run again.
// store all my test results
var results = [];
// Wrap the karma result function
var resultFunc = window.__karma__.result;
window.__karma__.result = function(result){
// run the original function
resultFunc(result);
// push each result on my storage array
results.push(result);
}
// wrap the karma complete function
var completeFunc = window.__karma__.complete;
window.__karma__.complete = function(result){
// run the original function
completeFunc(result);
// determine success
var success = results.every(function(r){ return r.success });
if (success) {
// display a success notification
}
else {
// display a test failure notification
}
// reset the result function
window.__karma__.result = resultFunc;
// reset the complete function
window.__karma__.complete = completeFunc;
}