We Keep Coding

How to skip or ignore programmatically a suite in CodeceptJS

As the test suite grows I need to be able to run something in BeforeSuite() which will connect to external suite and skip the suite if an external resource is unavailable.
Feature('External Server');
BeforeSuite((I) => {
// Check if server is available and skip all scenarios if it is not
});
Scenario('Login to the server', (I) => {
// Do not run this if the server is not available
})
I understand I could probably set a variable, but I think it would be nice if there was a way to tell the runner that a suite has been skipped.
The goal is to have a suite marked as skipped in the output eg:
Registration --
✓ Registration - pre-checks in 4479ms
✓ Registration - email validation in 15070ms
✓ Registration - password validation in 8194ms
External Server -- [SKIPPED]
- Login to the server [SKIPPED]

maybe prepend x before every scenario in your feature? example xScenario. I don't think codecept supports something similar to only for features. it currently works with scenarios only as far as I know.

you can use
Scenario.skip
in your step definition to skip a scenario.
Note: if any steps have been executed before skipping then it will still show it in the report
https://codecept.io/basics/#todo-test

My answer is compiled from a number of comments on the CodeceptJS github and stackoverflow. However, I can't recall the exact links or comments which helped me derive this solution, it's been at least a year, maybe two, since I started and have slowly modified this.
Edit: Found the github thread - https://github.com/codeceptjs/CodeceptJS/issues/661
Edit2: I wrote a post about "selective execution" (which avoids tagging unwanted tests with skip status) https://github.com/codeceptjs/CodeceptJS/issues/3544
I'll add a snippet at the bottom.
I'm on CodeceptJS 3.3.6
Define a hook file (eg: skip.js) and link it to your codeceptjs.conf.js file.
exports.config = {
...
plugins: {
skipHook: {
require: "../path/to/skip.js",
enabled: true,
}
}
...
}
The basic code in skip.js is
module.exports = function () {
event.dispatcher.on(event.test.before, function (test) {
const skipThisTest = decideSkip(test.tags);
if (skipThisTest) {
test.run = function skip() {
this.skip();
};
return;
}
});
};
I've defined a decision function decideSkip like so:
function decideSkip(testTags) {
if (!Array.isArray(testTags)) {
output.log(`Tags not an array, don't skip.`);
return false;
}
if (testTags.includes("#skip")) {
output.log(`Tags contain [#skip], should skip.`);
return true;
}
if (
process.env.APP_ENVIRONMENT !== "development" &&
testTags.includes("#stageSkip")
) {
output.log(`Tags contain [#stageSkip], should skip on staging.`);
return true;
}
}
(Mine is a bit more complicated, including evaluating whether a series of test case ids are in a provided list but this shows the essence. Obviously feel free to tweak as desired, the point is a boolean value returned to the defined event listener for event.test.before.)
Then, using BDD:
#skip #otherTags
Scenario: Some scenario
Given I do something first
When I do another thing
Then I see a result
Or standard test code:
const { I } = inject();
Feature("Some Feature");
Scenario("A scenario description #tag1 #skip #tag2", async () => {
console.log("Execute some code here");
});
I don't know if that will necessarily give you the exact terminal output you want External Server -- [SKIPPED]; however, it does notate the test as skipped in the terminal and report it accordingly in allure or xunit results; at least insofar as CodeceptJS skip() reports it.
For "selective execution" (which is related but not the same as "skip"), I've implemented a custom mocha.grep() utilization in my bootstrap. A key snippet is as follows. To be added either to a bootstrap anonymous function on codecept.conf.js or some similar related location.
const selective = ["tag1", "tag2"];
const re = selective.join("|");
const regex = new RegExp(re);
mocha.grep(regex);

Protractor repeat specs

I have a case like this
exports.config = {
seleniumAddress: 'http://localhost:4444/wd/hub',
specs: [
'test/scenarios/user/login.js',
'test/scenarios/user/choose_user_1.js',
'test/scenarios/user/change_user.js',
'test/scenarios/user/choose_user_2.js',
'test/scenarios/user/change_user.js',
'test/scenarios/user/choose_user_3.js',
'test/scenarios/user/logout.js'
]
}
But protractor doesn't reuse change_user.js more than once..
I have to create change_user_1.js and change_user_2.js to get what I want.. Is there a way to deactivate this behavior, or I should do my tests differently?
Best Regards

As far as i know, you cannot call same script twice. We had similar issue and here's what I did to fix it - Use jasmine-data-provider, create separate suites instead of scripts and loop through them using data provider. Here are the steps that i would follow -
Install jasmine-data-provider npm package.
Create two describe suites, one for choose_user and the other for change_user.
Pass multiple data to these describe suites using jasmine-data-provider.
Each time a choose_user - describe runs, a change_user - describe also runs next to that.
Here's a sample code -
var dp = require('../node_modules/jasmine-data-provider'); //Install the npm package and provide its path
//Data provider object to store data that script uses
var objectDataProvider = {
'Test1': {user1: 'user_1'},
'Test2': {user1: 'user_2'},
'Test3': {user1: 'user_3'},
};
//Jasmine Data Provider function automatically loops through the tests - Test1, Test2, Test3
dp(objectDataProvider, function (data) {
describe('choose_user Test:', function(){
//Choose User specs that's applicable for one user
//To use the objectDataProvider data use - data.user1 all the time
});
describe('change_user Test:', function(){
//Change User specs that's applicable for one user
});
});
This script should run choose_user and change_user specs 3 times and then you can continue execution with rest of the scripts in pipe.
Hope it helps.

Organizing code of a midly sized javascript client side app for testing

I'm building a midly size app using backbone and its friends jquery and underscore. My plan is to use QunitJS to create unittests.
I already have a Proof-Of-Concept of the app, so I basicly have a good grasp of how the code should look like without having to test it. It looks like that:
(function() {
// prepare some model
var Query = BackboneModel.extend({});
query = new Query();
// register some events
$('body.something').click(function() {
query.set('key', 'value');
# ...
});
// create some backbone view
var QueryView = Backbone.View.extend({...})
// init backbone view
query.view = new QueryView();
// add some plumbing here ...
// and so on...
})();
Otherwise said:
I wrap the module inside a function to avoid pollution
Class declaration and class use is interleaved
event registration is interleaved with the rest of the code
variables global to the module are reused inside the module
Now I need to test that. The problem is, I think, mainly about event registration and plumbing.
My plan is to wrap the code in functions and export every function and objects I want to test. The code will look like this:
var app = (function() {
var app = {}
// prepare some model
var Query = BackboneModel.extend({});
app.query = new Query();
// register some events
app.registerEvent = function() {
$('body.something').click(function() {
query.set('key', 'value');
# ...
});
};
app.registerEvent(); // XXX: call immediatly
// create some backbone view
app.QueryView = Backbone.View.extend({...})
// init backbone view
app.query.view = new QueryView();
// add some plumbing here ...
// wrapped in function with correct arguments and call it immediatly
// and so on...
// ...
return app;
})();
This is the first time I need to write tests in javascript for this kind of application so I'm wondering whether my approach to make the code testable is correct or can be improved. For instance, It seems silly to me to wrap the registration of events in function without arguments and call them immediatly.
Is there a javascript way to do this?

So I found an excellent way to test private functions while keeping my production code clean. I am not sure if you are using any build system like Grunt or Gulp, but if you are open to it you could do something like this:
//has a dependency of 'test' causing it to run testing suite first
gulp.task('js', ['test'], function() {
return gulp.src(source)
.pipe(plumber())
//create sourcemaps so console errors point to original file
.pipe(sourcemaps.init())
//strip out any code between comments
.pipe(stripCode({
start_comment: 'start-test',
end_comment: 'end-test'
}))
//combine all separatefiles into one
.pipe(concatenate('mimic.min.js'))
//minify and mangle
.pipe(uglify())
.pipe(sourcemaps.write('maps'))
.pipe(gulp.dest('dist/js'));
});
And the file could look like:
var app = (function () {
function somePrivateFunction () {}
function someotherPrivateFunction () {}
var app = {
publicFunction: function(){}
publicVar: publicVar
}
/* start-test */
app.test = {
testableFunction: somePrivateFunction
}
/* end-test */
}());
Everything between the test comments gets stripped out after the tests are run so the production code is clean. Grunt has a version of this and I assume any automated build system can do the same. You can even set a watch task so that it runs the tests on every save. Otherwise you'll have to manually remove the exported test object before deployment.
In the case of Backbone just attach a test object to the module and reference that object in the tests. Or if you really want to separate it, set the object in the global scope. window.testObject = { //list of objects and methods to test... }; and strip that code out before deployment.

Basically what I do is avoid any calls in the library I want to test. So everything is wrapped in a function and exported.
Then I have two other files one is main.js where I do the plumbing and should be tested using integration tests with selenium and tests.js which does unit testing.

Unit Testing Assertion when using Mocks

I am writing a unit test using mocks and would like to ask an opinion. My specific example is mocking the "window" object, however the problem could occur in other scenarios.
To note, I am reverse engineering the unit tests, TDD is a new style which we are adopting here.
The module I am testing is one that will return the hash part of the URL and grab parameters from it. For example if the hash value was:
#param=value&param2=value2
Then the module would return an object like:
{
param: 'value',
param2: 'value2'
}
The first unit test I am writing is one which returns the hash value without the hash. Here is the method:
getCurrentHash: function() {
return window.location.hash.slice(1);
},
I have mocked the window object to set window.location.hash equal to #param=value&param2=value2
I am expecting getCurrentHash to return param=value&param2=value2 in this particular instance.
The window object is a dependency on the hash module, I have switched the path when running the tests to use the mocked version.
The Question
Should unit test explicitly test against the mocked data like this:
tdd.test('getCurrentHash', function () {
var currentHash = hash.getCurrentHash();
currentHash.should.equal('param=value&param2=value2');
});
Or should the unit test be testing the format of the return value, like this:
tdd.test('getCurrentHash', function () {
var currentHash = hash.getCurrentHash();
currentHash.should.not.contain('#');
currentHash.should.be.a('string');
currentHash.should.have.length.above(0);
});
This question came about due to the fact that the mock might change. If for example we decided to add spaces into our mock we would end up amending it to param=value&param2=value2+withspace. This would then make the test fail and I would need to change the test to:
tdd.test('getCurrentHash', function () {
var currentHash = hash.getCurrentHash();
currentHash.should.equal('param=value&param2=value2+withspace');
});
However, in the case of the second test which is more generic, I would not need to change the unit test.
I have read a few articles including the ones listed below, but I couldn't find any detailed reasoning on what an assertion should assert
http://martinfowler.com/articles/mocksArentStubs.html
Unit Tests: How to Assert? Asserting results returned or that a method was called on a mock?

you want to test the behaviour you want to see from the thing you are testing. Your behaviour is (based on the implementation) that the window locations hash value minus the '#' char is returned. So you should test things like:
when no has values exist, nothing is returned (but no error is returned (unless you explicitly want that))
The expected value is returned when a value exists
your second test has little value IMHO as I could change the implementation to this and the tests would still pass.
getCurrentHash: function() {
return "bob";
}
This makes the test fairly useless. your tests exist to verify the behaviour of the system. So write tests that set up happy path conditions, failure cases and edge cases and have them verify that things behave in the way you expect.

QUnit multiple scripts in one page but no interaction between them

I'm very new to unit testing (this is my first day working with QUnit and I've never worked with anything other testng system before), and I'm a bit unclear on how to test stuff from multiple script files in one QUnit page without letting the scripts interact with each other. What I mean is, say, if I have script1.js, and it calls hello(), and hello() is defined in script2.js, how can I run a unit test on script1.js to make sure it calls hello(), but mock the output of hello() so that it's a true unit test, and then run script2.js's hello().
Basically, how am I supposed to hide one script's global variables and functions from another script in a single QUnit page?

This depends entirely on how the various script files are organized as well as the system as a whole. If you were using Angular, for example, then you are able to inject dependencies when you include a module in another script file. There are tools for mocking out things and "spying" on function calls such as Sinon, but it still depends heavily on how your code is organized.
For the sake of argument, let's say the two files look like so, and we'll ignore the design pattern (although you should seriously be considering that)...
// File A:
window.greeting = function() {
var world = hello();
return 'hello ' + world;
}
// File B:
window.hello = function() {
// possibly lots of code to determine what to return...
var value = 'foobar';
return value;
}
The hello() function could just as easily return any other value based on the state of the system, user input, etc. For our case it doesn't, and what we want to do is mock out File B's code so that we don't have to test what it is doing, just that it gives back a string. We could (and should) do this with a proper mocking/dependency injection library. However, just to give you a sense of the minimal setup you could do, and to see the general approach, here's our QUnit test file:
var _hello;
QUnit.module('File A', {
setup: function() {
_hello = window.hello; // hold onto the old value
// now we mock out hello()
window.hello = function() {
window.hello.called++; // track calls to it
return 'world'; // return our static value
}
window.hello.called = 0;
},
teardown: function() {
// put the old one back
window.hello = _hello || window.hello;
}
});
QUnit.test('Ensure greeting is correct', function(assert) {
var result = greeting();
assert.equal(window.hello.called, 1, 'hello should be called only once');
assert.equal(result, 'hello world', 'The greeting call should be "hello world"');
});
And if you want to see it running, here is a jsfiddle for you. As I said, this is a simple example to show you how you could do this, but you should look into proper code organization (think AMD modules, require, Angular, Ember, things like this) and a proper mocking library.