We Keep Coding

Angular/ Protractor E2E Test fails when run in Azure DevOps Pipeline. Passes locally

I am trying to run e2e tests on my angular application. My tests pass locally when I run ng e2e but not in my pipeline. I'm going to share my protractor.conf, the pipeline tasks and the output I get from the task that fails below.
Some more background:
I am trying to run e2e code from an otherwise empty angular app. The
test I am trying to run simply logs into AD by navigating to my
website, entering a username/ password and then checking the that the
user is redirected to my website.
I am running this from a release
pipeline where the repo containing the e2e tests is added as an
artifact.
I am using xpaths to find elements on the page
I've omitted my jasmine code because I do not think it is relevant since the tests pass locally. However, if I'm wrong on this point let me know and I will post it.
protractor.conf.js
const { SpecReporter } = require('jasmine-spec-reporter');
process.env.CHROME_BIN = process.env.CHROME_BIN || require("puppeteer").executablePath();
exports.config = {
allScriptsTimeout: 11000,
specs: [
'./src/**/*.e2e-spec.ts'
],
capabilities: {
chromeOptions: {
args: ["--headless", "--disable-gpu", "--window-size=1200,900"],
binary: process.env.CHROME_BIN
},
'browserName': 'chrome'
},
directConnect: true,
baseUrl: 'http://localhost:4200/',
framework: 'jasmine',
jasmineNodeOpts: {
showColors: true,
defaultTimeoutInterval: 120000,
print: function() {}
},
onPrepare() {
require('ts-node').register({
project: require('path').join(__dirname, './tsconfig.e2e.json')
});
jasmine.getEnv().addReporter(new SpecReporter({ spec: { displayStacktrace: true } }));
}
};
Pipeline Tasks:
Failed Task Output:
Please help me get these tests to pass in azure
Thanks!
EDIT: I have that 'Update Webdriver' task because I read I should do it somewhere, it doesn't actually change the outcome if it is there or not

Have you tried to increase your allScriptsTimeout?
The timeout in milliseconds for each script run on the browser. This should be longer than the maximum time your application needs to stabilize between tasks.

Why multiCapabilities don't accept specs files from spec:[], why I need to divide specs in each instance in multiCapabilities

I am trying to distribute specs[file1, file2, file3] in multiCapabilities but then all the instances start running all the spec files, while I just want all instances to pic specs and test once.
When I pass specs inside multiCapabilities then it works fine but if I keep specs[] out of multiCapabilites and use shardTestFiles: true, then all browser instances run all the specs rather than all browser instances should rest all the specs once only.
exports.config = {
shardTestFiles: true,
maxSessions: 5,
specs: ['./Tests/spec1.js','./Tests/spec2.js', './Tests/spec3.js' ],
multiCapabilities: [
{
browserName : 'chrome',
seleniumAddress: 'http://localhost:4444/wd/hub'
},
{
browserName : 'chrome',
seleniumAddress: 'http://172.23.230.198:4444/wd/hub'
}
]
}
Expected Result:
2 instances of chrome browser should open and run all the 3 specs and share the result.
Actual Result:
2 instance of chrome browser open and run 3 specs in each browser repeatedly while 2 instances should run 3 specs once only.

angularjs protractor 3.3.0 not reporting specs

I don't see expected output for a passed run, the assertions are not listed. I expect to see the assertion in this line "1 spec, 0 failures".
The output:
[18:28:06] I/hosted - Using the selenium server at http://localhost:4444/wd/hub
[18:28:06] I/launcher - Running 1 instances of WebDriver
Started
.
1 spec, 0 failures
Finished in 0.854 seconds
[18:28:08] I/launcher - 0 instance(s) of WebDriver still running
[18:28:08] I/launcher - chrome #01 passed
Expected end of run output as seen on protractor's web site, http://www.protractortest.org/#/ "Run the test"):
1 test, 3 assertions, 0 failures
The spec:
describe('Viewing index.html', function() {
'use strict';
beforeEach(function(){
browser.get('/');
});
it('should have pages in left nav', function() {
expect(element.all(by.repeater('page in adminClient.selectedSite.pages')).count()).toBeGreaterThan(0);
});
});
I verified that the by.repeater locator worked:
element.all(by.repeater('page in adminClient.selectedSite.pages')).count()
.then(function(count){
console.log('page count: ' + count);
});
[UPDATE] According to this SO, it is a version issue and there is a recommendation to inject jasmine reporters on the onPrepare hook but that created more runtime errors for me.
stack overflow question
My protractor config:
exports.config = {
allScriptsTimeout: 11000,
chromeOnly: true,
chromeDriver: 'node_modules/protractor/bin/selenium/chromedriver_2.21',
seleniumAddress: 'http://localhost:4444/wd/hub',
specs: ['tests/e2e/*-spec.js'],
capabilities: {
'browserName': 'chrome'
},
baseUrl: 'http://localhost:8889/',
framework: 'jasmine',
jasmineNodeOpts: {
showColors: true,
defaultTimeoutInterval: 30000
}
};

To see the spec names and assertions you must pass the --verbose flag to protractor. If you are using grunt or something to run protractor, you'll need to specify this flag in your config.
EDIT
After reading your edit I believe I have found the solution to your issue. I've tried it with a project of my own and it appears to work.
The underlying problem is that you are likely using protractor 3, which no longer supports many of the previous options particularly within jasmineNodeOpts. To correct this issue, you should downgrade your version of protractor to 2, the latest version is 2.5.1
Here's the related issue on protractor's github repository. It also mentions a custom reporter within the onPrepare hook like you were talking about, but a different one: jasmine-spec-reporter. I got that working as well with a slightly different configuration than what you are using but it does not display the assertions, just has a much better output for the tests, which I quite like:
jasmineNodeOpts: {
print: function () {} // remove dots for each test
},
onPrepare: function () {
var SpecReporter = require('jasmine-spec-reporter');
jasmine.getEnv().addReporter(new SpecReporter({displayStackTrace: true}));
}

Configuring multiple capabilities with promises

This is a follow-up to the Set firefox profile with protractor topic.
According to the setFirefoxProfile howto, it is possible to set a firefox profile with a special "helper" js code which uses firefox-profile and q libraries to make an encoded firefox profile on the fly.
This worked for me until I've tried to use multiple browsers and configuring multiCapabilities:
exports.config = {
seleniumAddress: 'http://localhost:4444/wd/hub',
multiCapabilities: [
{
browserName: 'chrome',
specs: [
'footer.disabledCookies.spec.js'
],
chromeOptions: {
prefs: {
'profile.default_content_settings.cookies': 2
}
}
},
...
// other capabilities here
...
helper.getFirefoxProfile()
},
...
}
With this setup I'm getting an error (full traceback here):
Spec patterns did not match any files.
As I understand, this means that the setup with firefox profile is missing specs key. In other words, it cannot find any tests to run.
I've tried to include specs into the capabilities dictionary inside the helper itself, but the error persists.
How to fix the error and set firefox profile if using multiCapabilities?
As a workaround, I've created a separate protractor configuration file with only firefox configured (using capabilities) and set grunt to run protractor twice - one for this "firefox with a profile" config and the other one for all other browsers.

Right now, protractor can only accept promise as capabilities if we are NOT using multicapabilities. The reason for this is because multiCapabilities runs each task in a new process, so the promise (function) cannot be passed (single capabilities work because we're not forking).
Alternatively we could resolve capabilities in the launcher, before passing the resolved capabilities into the new processes; however, this will break the ability to set up proxies (https://github.com/angular/protractor/pull/1040), which relies on capability promises to be resolved after driverProvider setup.
I can't think of an easy way of doing this (without large refactoring), but it is definitely doable.
I created an issue for Protractor (https://github.com/angular/protractor/issues/1594). Please follow that and/or comment on it if this is something you need or you have other ideas to implement it.
For now you would need to use the workaround you mentioned in your original question.
UPDATE
https://github.com/angular/protractor/pull/1629 supports this. Starting in protractor 1.6 (or if you sync to master) you can pass in a function to config.getMultiCapabilities like onPrepare and onCleanup. This function can return a promise to multiCapabilties (i.e. array of capabilities).
See https://github.com/angular/protractor/blob/master/spec/getCapabilitiesConf.js for an example.

Following the pull request sent by #hankduan, here is how have I used getMultiCapabilities() to combine different capabilities where one of them is a promise (needed for firefox-profile to be set):
"use strict";
var FirefoxProfile = require("firefox-profile");
var q = require("q");
exports.config = {
seleniumAddress: "http://127.0.0.1:4444/wd/hub",
getMultiCapabilities: function() {
var deferred = q.defer();
var multiCapabilities = [
{
browserName: "chrome",
specs: [
"footer.disabledCookies.spec.js"
],
chromeOptions: {
prefs: {
"profile.default_content_settings.cookies": 2
}
}
},
{
browserName: "chrome",
specs: [
"*.spec.js"
],
exclude: [
"footer.disabledCookies.spec.js",
"footer.disabledJavascript.spec.js",
"footer.disabledFlash.spec.js"
]
},
{
browserName: "chrome",
specs: [
"footer.disabledFlash.spec.js"
],
chromeOptions: {
args: [
"--disable-internal-flash",
"--disable-bundled-ppapi-flash",
"--disable-plugins-discovery"
]
}
}
];
// Wait for a server to be ready or get capabilities asynchronously.
setTimeout(function() {
var firefoxProfile = new FirefoxProfile();
firefoxProfile.setPreference("javascript.enabled", false);
firefoxProfile.encoded(function (encodedProfile) {
var capabilities = {
"browserName": "firefox",
"firefox_profile": encodedProfile,
"specs": [
"footer.disabledJavascript.spec.js"
]
};
multiCapabilities.push(capabilities);
deferred.resolve(multiCapabilities);
});
}, 1000);
return deferred.promise;
},
...
};
Hope this would help somebody in the future.

Is it possible to add a plugin to chromedriver under a protractor test?

I've been trying to handle the basic authentication during my protractor test. Some hard time on it, so i've found a chrome plugin wich sends automatically my credentials for websites that require basic authentication.
As each time that chromedriver is executed, a new profile is loaded, how can i permanelty add a plugin to my tests? I know that there is https://sites.google.com/a/chromium.org/chromedriver/extensions, but i dont think this very clear.

You need to configure extensions list inside chromeOptions:
capabilities {
'browserName': 'chrome',
'chromeOptions': {
'extensions': ['base64 encoded extension']
}
}
Note that it in extensions, it is important to provide a list of base-64 encoded packed Chrome extension.
To get a base64 encoded extension, you need to read the .ctx extension file and encode the contents with base64. For example, using python:
>>> import base64
>>> data = open('path_to_the_ctx_extension').read()
>>> base64.standard_b64encode(data).decode('UTF-8')
# outputs the encoded chrome extension which you can paste in the config
Or, easier, make a helper.js file using fs and q:
var q = require('q');
var fs = require('fs');
exports.getCapabilities = function (filename) {
var deferred = q.defer();
fs.readFile(filename, function (err, data) {
var capabilities = {
'browserName': 'chrome',
'chromeOptions': {
extensions: [
data.toString('base64')
]
}
};
deferred.resolve(capabilities);
});
return deferred.promise;
};
Then, in your protractor config, use this getCapabilities() function to get capabilities:
var helper = require('./helper.js');
exports.config = {
capabilities: helper.getCapabilities('/path/to/crx/extension'),
...
}
Currently, it works with a single extension, so there is a room for improvement.
Also, look through the following issue in case you have problems:
Setting Chrome Options

Check this: https://github.com/andresdominguez/elementor/blob/master/bin/elementexplorer.js#L194
Here I am loading an extension from a local directory. The extension is not a crx file, but the uncompressed version.
'chromeOptions': {
'args': ['--load-extension=' + extensionPath]
}

Instead of committing the extension with your code and having to load it from disk when you run the tests you might want to consider using the authenticator-browser-extension Node module I have recently open-sourced.
To use the module install it from npm:
npm install --save-dev authenticator-browser-extension
And import in the protractor.conf.js:
const { Authenticator } = require('authenticator-browser-extension');
exports.config = {
capabilities: {
browserName: 'chrome',
chromeOptions: {
extensions: [
Authenticator.for('username', 'password').asBase64()
]
}
},
}
Pro tip: remember not to commit your credentials with your code, consider using env variables instead.
Hope this helps!
Jan