The Problem:
We have a rather large test codebase. From time to time, instead of executing all the tests, we execute them individually or in packs. But, sometimes, we see the unexpected test failures because of the tests being interconnected, coupled. For example, one test assumes there is some data created by a previous test - running this kind of test individually will fail.
The Question:
Is it possible to automatically detect which Protractor tests are coupled in the project?
Our current idea is to somehow randomize the test execution order or randomly pick up a pack of tests from all the available tests and check if there are no failures. Hence, the other question: is it possible to change/randomize the Protractor test discovery and change the order of test execution?
Inspired by the Ned Batchelder's "Finding test coupling" blogpost and the Python nose test runner's nose-randomly plugin:
Randomness in testing can be quite powerful to discover hidden flaws
in the tests themselves, as well as giving a little more coverage to
your system.
By randomly ordering the tests, the risk of surprising inter-test
dependencies is reduced - a technique used in many places, for example
Google’s C++ test runner googletest.
You can run tests randomly (at the file level) by setting the random property in your config . You can also set your salt/seed so it's reproducibly random.
/**
* If true, run specs in semi-random order
*/
random?: boolean,
/**
* Set the randomization seed if randomization is turned on
*/
seed?: string,
You could also turn on shardTestFiles (parallel test runs), which should also be very telling in how coupled your tests are.
Did you try shuffling "it" blocks like below:
var shuffle = function (items) {
var item, randomIndex;
for(var i = 0; i < items.length; i++){
randomIndex= (Math.random() * items.length) | 0;
item = items[i];
items[i] = items[randomIndex];
items[randomIndex] = item;
}
}
describe('Suite', function() {
it("should a", function () {
console.log("execute a");
});
it("should b", function () {
console.log("execute b");
});
it("should c", function () {
console.log("execute c");
});
shuffle(this.children); // shuffle the 'it' blocks
});
Source: Can protractor tests be run in a random order?
One problem is you likely have no idea how tests might be coupled. If one test referenced some variables from another test, you might be able to find those automatically but that's only one way tests might be coupled and probably not a likely scenario.
My first thought was to just run them individually and see which ones fail. The problem is that if you aren't cleaning state between tests, you might change the order (randomizing them, as you suggested) but if test 50 expects the data that test 20 set up but in the new order, test 20 still runs before test 50... test 50 will still pass. You will find some but probably not all until you run all of them in a random order several times.
You don't describe your application but my second thought was if there was a way to get back to a clean slate between tests, you should be able to find the tests that rely on other tests to set up data. I'm kinda surprised you aren't doing that already but if there's a long setup process that has to run to install a clean slate, etc. that might be an issue. Depending on your system, you might be able to snapshot a VM after a clean install and restore it to "quickly" get back to clean or you may be able to roll back SQL tables, etc. It really depends on your system and without more details on the system, it's hard to offer advice.
Another option is to go to those that wrote or maintain the tests and have them self-identify the tests they own that are coupled and fix them. This likely won't find them all but it might be a semi-quick start.
Oh... I just thought of one more thing. If you could reverse the order of test execution, that should be better than randomizing the execution. With reverse order, NO test would run after it's former predecessor and you should be able to find them all in one go.
Related
I noticed the default behavior of Karma for my Angular builds is to run the Jasmine unit tests in random order. What is the benefit of running your tests in random order, vs running them in the same order every time?
Sometimes tests modify state that doesn't get reset for every test run. Maybe a test modifies a global variable that is shared by all tests. Maybe a test writew something to the database and doesn't clean it up when the test is done. These things shouldn't happen, but sometimes they do happen.
For example:
// Stupid contrived example. Don't ever do this.
let num = 0
it('A', () => {
expect(num + 1).toEqual(1)
})
it('B', () => {
num = 10
expect(num + 2).toEqual(12)
})
it('C', () => {
expect(num + 3).toEqual(13) // passes if B is run before C, otherwise fails
})
This may work fine if you always run test A, then B, then C. But then strange things my happen if you run only test C. Because of the changes that test A and B made, C passes when you run them all together. But then you run test C alone and it fails.
And now you are staring at you terminal output dumbfounded muttering "how on earth can test C fail?! It just passed!"
Randomizing the order helps with this a bit. Each test should be totally isolated and it shouldn't matter what order they run in. Randomizing them may help uncover this badness.
So, given that the order shouldn't matter, then why not randomize the order?
I am trying to understand how to prevent memory leaks when using timers. I believe the Chrome Dev Tools Performance tab (I am still learning about this feature) memory graph is showing me a bad pattern in terms of memory management. I believe it is a case of a "sawtooth" pattern each time a Timer is fired.
I have a simple test case (1. right click 'timer-adder.html', 2. click 'Preview' to get a preview of the loaded HTML) that involves using a constructor function to create objects that work as timers, in other words for each update, the DOM is changed inside a setInterval callback.
//...
start: function (initialTime, prefixedId) {
let $display = document.getElementById(prefixedId + '-display');
if ($display.style.display === 'none') { $display.style.display = 'block'; }
$display.textContent = TimerHandler.cycle(initialTime);
$display = null;
return setInterval(function () {
this.initialTime--;
(this.initialTime === 0 && this.pause());
document.getElementById(prefixedId + '-display').textContent = TimerHandler.cycle(this.initialTime);
}.bind(this), 1000);
},
/...
Attempts to minimize leakage although not directly related to what I believe to be the problem at hand:
assigning $display to null to make it collectible (garbage collection).
clearing the interval;
What I identify as a bad pattern:
View post on imgur.com (zoomable)
A fair memory usage would translate to DevTools showing a horizontal line? What would be a safer approach to this side effect? Because, as the test is right now, I think that in the long run, multiple active timers would overload memory and thus result in a noticeable decrease in performance.
PS: As a newcomer, I hope this is a fair presentation of the problem. Thank you.
As you can see from the last part, the Garbagge Collector was able to free that memory. Thus there is no memory leaking.
Garbagge Collection is a complicated task, thats why V8 tries to minimize the number of stop-the-world collections. Or in other words: It only cleans up memory if it needs it.
In your case it doesn't. There is still enough space available.
If (a noticible amount of) memory persists across GC calls, then you do have a memory leak.
assigning $display to null to make it collectible (garbage collection).
Thats just unneccessary. $display is not accessed in the inner function, therefore it is viable for gc'ing after init executed. Reassigning it doesn't change that.
Years ago, I heard about a nice 404 page and implemented a copy.
In working with ReactJS, the same idea is intended to be implemented, but it is slow and jerky in its motion, and after a while Chrome gives it an "unresponsive script" warning, pinpointed to line 1226, "var position = index % repeated_tokens.length;", with a few hundred milliseconds' delay between successive calls. The script consistently goes beyond an unresponsive page to bringing a computer to its knees.
Obviously, they're not the same implementation, although the ReactJS version is derived from the "I am not using jQuery yet" version. But beyond that, why is it bogging? Am I making a deep stack of closures? Why is the ReactJS port slower than the bare JavaScript original?
In both cases the work is driven by minor arithmetic and there is nothing particularly interesting about the code or what it is doing.
--UPDATE--
I see I've gotten a downvote and three close votes...
This appears to have gotten response from people who are (a) saying something sensible and (b) contradicting what Pete Hunt and other people have said.
What is claimed, among other things, by Hunt and Facebook's ReactJS video, is that the synthetic DOM is lightning-fast, enough to pull 60 frames per second on a non-JIT iPhone. And they've left an optimization hook to say "Ignore this portion of the DOM in your fast comparison," which I've used elsewhere to disclaim jurisdiction of a non-ReactJS widget.
#EdBallot's suggestion that it's "an extreme (and unnecessary) amount of work to create and render an element, and do a single document.getElementById. Now I'm factoring out that last bit; DOM manipulation is slow. But the responses here are hard to reconcile with what Facebook has been saying about performant ReactJS. There is a "Crunch all you want; we'll make more" attitude about (theoretically) throwing away the DOM and making a new one, which is lightning-fast because it's done in memory without talking to the real DOM.
In many cases I want something more surgical and can attempt to change the smallest area possible, but the letter and spirit of ReactJS videos I've seen is squarely in the spirit of "Crunch all you want; we'll make more."
Off to trying suggested edits to see what they will do...
I didn't look at all the code, but for starters, this is rather inefficient
var update = function() {
React.render(React.createElement(Pragmatometer, null),
document.getElementById('main'));
for(var instance in CKEDITOR.instances) {
CKEDITOR.instances[instance].updateElement();
}
save('Scratchpad', document.getElementById('scratchpad').value);
};
var update_interval = setInterval(update, 100);
It is doing an extreme (and unnecessary) amount of work and it is being done every 100ms. Among other things, it is calling:
React.createElement
React.render
document.getElementById
Probably with the amount of JS objects being created and released, your update function plus garbage collection is taking longer than 100ms, effectively taking the computer to its knees and lower.
At the very least, I'd recommend caching as much as you can outside of the interval callback. Also no need to call React.render multiple times. Once it is rendered into the dom, use setProps or forceUpdate to cause it to render changes.
Here's an example of what I mean:
var mainComponent = React.createElement(Pragmatometer, null);
React.render(mainComponent,
document.getElementById('main'));
var update = function() {
mainComponent.forceUpdate();
for(var instance in CKEDITOR.instances) {
CKEDITOR.instances[instance].updateElement();
}
save('Scratchpad', document.getElementById('scratchpad').value);
};
var update_interval = setInterval(update, 100);
Beyond that, I'd also recommend moving the setInterval code into whatever React component is rendering that stuff (the Scratchpad component?).
A final comment: one of the downsides of using setInterval is that it doesn't wait for the callback function to complete before queuing up the next callback. An alternative is to use setTimeout with the callback setting up the next setTimeout, like this
var update = function() {
// do some stuff
// update is done to setup the next timeout
setTimeout(update, 100);
};
setTimeout(update, 100);
I've searched the site and some literature and couldn't get to a clear answer. I'm trying to learn unittesting while constructing a new webpage that simply works as a whiteboard to which you can add post-its.
I have a Canvas object which represents the whiteboard, and a ticket object to represent the post-its. I have (for now) global function to retrieve the one and only canvas, which i test like this:
this.testRetrieveCanvas = function()
{
var canvas = getCanvas();
this.assertTrue( canvas != null );
}
this.testCanvasType = function()
{
var canvas = getCanvas();
this.assertTrue( canvas instanceof Canvas );
}
this.testIfCanvasIsReused = function()
{
var canvas = getCanvas();
this.assertEquals( canvas, getCanvas() );
}
So, i test for three things:
Does the method return a canvas?
Is it an acutal canvas?
Does the method give me the same canvas always?
No problems so far. But a little later, i'm testing "adding the ticket to the canvas":
this.testAddTicketToCanvas = function()
{
var ticket = factory.createTicket("yellow");
var canvas = getCanvas();
canvas.addTicket( ticket );
this.assertTrue( canvas.contains( ticket ) );
};
As you can see, i'm using the getCanvas() function inside my test. Is this a dependent test now? I mean, the first three tests have to pass, if i want this test to be able to run without doubts. If it is dependent, how would i fix this?
Strictly speaking, you should override getCanvas() to return a preconstructed (i.e. the original constructor isn't called) partial mock of canvas. With that said, if canvas' constructor is an empty function and the getCanvas method has no business logic involved, you shouldn't have a problem.
I'd be more wary of the last two statements used together. canvas.addTicket( ticket ); is ok, since it's the function being tested. but then you're asserting that you have added the ticket using a method from the same object. What if this method isn't implemented yet, or returns false, or worse true? What if your addTicket method has secondary effects that could make it add the ticket to the list but change a flag that makes contains() throw an exception, or return false, or true? What if contains has certain business logic that makes it return false for the ticket your sending it, but true for the same ticket in production environment (i.e. your test ticket hasn't been properly initialized, is missing a state, has been marked as excluded from the environment's business flow), what if it has no logic now, but two months into the project the logic changes so that your test fails but all else works (a new state is added, and objects without this state are deemed non existant, except for clients a, b and c). I could go on.
My point is without the code we can't specifically answer your question, only give you pointers and general answers like the one above. If you really don't want to post code, then take into consideration all of these scenarios, and all the other scenarios you can think of, and if under these scenarios testing your code this way won't break code nor tests now and for the foreseeable future, then you're ok.
The short answer to your question is, no.
If your code does not violate any unit testing
principles as long as you adhere to these rules.
Usually, with those questions there is a huge discussion about mocking certain parts of your tests e.g. your getCanvas() function. I agree that there is reason behind this discussion and that if you want to proceed with testing or TDD in general you should dive deeper into this topic. (Please refer to this excellent article from Martin Fowler).
However, for the question if this is a valid unit test, I think it is not relevant as long as you adhere to the unit test rules.
Disclaimer - I've tried finding an answer to this via google/stackoverflow, but I don't know how to define the problem (I don't know the proper term)
I have many small AI snippets such as what follows. There is an ._ai snippet (like below) per enemy type, with one function next() which is called by the finite state machine in the main game loop (fyi: the next function doesn't get called every update iteration, only when the enemy is shifted from the queue).
The question: How do I test every case (taking into account some enemy AI snippets might be more complex, having cases that may occur 1 in 1000 turns) and ensure the code is valid?
In the example below, if I added the line blabla/1 under count++, the error might not crop for a long time, as the Javascript interpreter won't catch the error until it hits that particular path. In compiled languages, adding garbage such as blabla/1 would be caught at compile time.
// AI Snippet
this._ai = (function(commands){
var count = 0;
return {
next: function(onDone, goodies, baddies) {
// If the internal counter reaches
// 2, launch a super attack and
// reset the count
if(count >= 2) {
commands.super(onDone);
count = 0;
}
else {
// If not performing the super attack
// there is a 50% chance of calling
// the `attack` command
if(chance(50)) {
var target = goodies[0];
commands.attack(onDone, target);
}
// Or a 50% chance of calling the
// `charge` command
else {
commands.charge(onDone);
count++;
}
}
}
};
})(this._commands);
I could rig the random generator to return a table of values from 0-n and run next 1000's of times against each number. I just don't feel like that is will concretely tell me every path is error free.
As you say, unit tests must test every path so you will be sure all works well.
But you should be able to decide which path the method will follow before calling it on your tests, so you're be able to know if the method behaviour is the expected one, and if there is any error.
So, for example, if there is a path that will be followed in only one of every 1000 executions, you shouldn't need to test all 0, 1, 2 ... 999 cases. You only one combination of results that behave distinctly.
For example, in the snippet shown you have these cases:
the counter has reached 2
the counter has not reached 2 and chance returns true
the counter has not reached 2 and chance returns false
One way to archieve this is taking control of the counter and of the chance method by mocking them.
If you want to know what happens when the counter has reached 2 and the next method is called, just pass a counter with 2 and call next. You don't need to reach 2 on the counter by really passing for all the code.
As for the randomizer, you don't need to try until the randomizer returns the value you want to test. Make it a mock and configure it to behave as you need for each case.
I hope this helps.