I want to be able to catch any and all failures within my specs and log it into my Test Rail integration. I have this pattern currently where I wrap all the tests in a try/catch and on the CATCH, I mark the test as FAILURE and add it to the results array. Inside the afterAll it will then send all these results to Test Rail's API.
This all works mostly correct. However, occasionally one of my tests will fail but will NOT trigger the catch part of the code. My spec file will proceed all the way to the done() and finish the full test and THEN will show my failure once the code gets to the AfterAll section.
I assume this is because the expect(lookForSomeObject) is still processing/looking in the background and it doesn't finally finish doing its thing until the spec file gets to the AfterAll section.
How can I update this pattern to be better.
afterAll(async () => {
var r = await tr.addResultsForCases(testCases);
});
it('Verifies Search', async (done) => {
var testCaseInfo: AddResultForCase = {
case_id: 75
}
try {
const locatorSearchBarInput = await loc.locatorSearchBarInput();
await waitToBeDisplayed(locatorSearchBarInput);
await locatorSearchBarInput.sendKeys('35011');
const locatorSearchButton = await loc.locatorSearchButton();
await waitToBeDisplayed(locatorSearchButton);
await click(locatorSearchButton);
await expect(await loc.locatorSearchBarText()).toBeDisplayed();
testCaseInfo.status_id = TestRailStatus.Passed;
testCases.results.push(testCaseInfo);
done();
} catch (e) {
testCaseInfo.status_id = TestRailStatus.Failed;
testCases.results.push(testCaseInfo);
done.fail(e);
}
});
I would get rid of try/catch
Add case_id into your it description at the env like so Verifies Search #75
Create your own custom jasmine reporter as described here https://jasmine.github.io/tutorials/custom_reporter
register it within your onPrepare()
specDone function will take spec parameter (or result, whatever you call it). Then add this snippet
specDone: function(spec) {
if (spec.status === 'passed') {
// do anything on success
} else if (spec.status === 'failed') {
let case_id = spec.description.match(/#([^#]*$)/)[1];
// report your case_id as failed
} else {
// block for skipped test cases (marked by `xit` or `xdescribe`)
}
}
That's it. No need to add your handling to each test case anymore.
Related
I want to test the execution of both fuctionUT and an inner async unwaited function externalCall passed by injection. The following code is simple working example of my functions and their usage:
const sleep = async (ms) => new Promise( (accept) => setTimeout(() => accept(), ms) )
const callToExternaService = async () => sleep(1000)
const backgroundJob = async (externalCall) => {
await sleep(500) // Simulate in app work
await externalCall() // Simulate external call
console.log('bk job done')
return 'job done'
}
const appDeps = {
externalService: callToExternaService
}
const functionUT = async (deps) => {
await sleep(30) // Simulate func work
// await backgroundJob(deps.externalService) // This make test work but slow down functionUT execution
backgroundJob(deps.externalService) // I don't want to wait for performance reason
.then( () => console.log('bk job ok') )
.catch( () => console.log('bk job error') )
return 'done'
}
functionUT( appDeps )
.then( (result) => console.log(result) )
.catch( err => console.log(err) )
module.exports = {
functionUT
}
Here there is a simple jest test case that fail but just for timing reasons:
const { functionUT } = require('./index')
describe('test', () => {
it('should pass', async () => {
const externaServiceMock = jest.fn()
const fakeDeps = {
externalService: externaServiceMock
}
const result = await functionUT(fakeDeps)
expect(result).toBe('done')
expect(externaServiceMock).toBeCalledTimes(1) //Here fail but just for timing reasons
})
})
What is the correct way to test the calling of externaServiceMock (make the test pass) without slowdown the performance of the functionUT ?
I have already found similar requests, but they threat only a simplified version of the problem.
how to test an embedded async call
You can't test for the callToExternaService to be called "somewhen later" indeed.
You can however mock backgroundJob and test that is was called with the expected arguments (before functionUT completes), as well as unit test backgroundJob on its own.
If a promise exists but cannot be reached in a place that relies on its settlement, this is a potential design problem. A module that does asynchronous side effects on imports is another problem. Both concerns affect testability, also they can affect the application if the way it works changes.
Considering there's a promise, you have an option to chain or not chain it in a specific place. This doesn't mean it should be thrown away. In this specific case it can be possibly returned from a function that doesn't chain it.
A common way to do this is to preserve a promise at every point in case it's needed later, at least for testing purposes, but probably for clean shutdown, extending, etc.
const functionUT = async (deps) => {
await sleep(30) // Simulate func work
return {
status: 'done',
backgroundJob: backgroundJob(deps.externalService)...
};
}
const initialization = functionUT( appDeps )...
module.exports = {
functionUT,
initialization
}
In this form it's supposed to be tested like:
beforeAll(async () => {
let result = await initialization;
await result.backgroundJob;
});
...
let result = await functionUT(fakeDeps);
expect(result.status).toBe('done')
await result.backgroundJob;
expect(externaServiceMock).toBeCalledTimes(1);
Not waiting for initialization can result in open handler if test suite is short enough and cause a reasonable warning from Jest.
The test can be made faster by using Jest fake timers in right places together with flush-promises.
functionUT( appDeps ) call can be extracted from the module to cause a side effect only in the place where it's needed, e.g. in entry point. This way it won't interfere with the rest of tests that use this module. Also at least some functions can be extracted to their own modules to be mockable and improve testability (backgroundJob, as another answer suggests) because they cannot be mocked separately when they are declared in the same module the way they are.
So basically I have some helpers method to help me debug my protractor test cases, one of my main ones is tho wait for an element to be clickable, I'm creating this loop to give the necessary time for protractor to find and make sure the element is enabled, but when an element is not found, either by the element not being found or a typo on my scrip, I would like the test run to STOP and mark it as a FAILURE..
async WaitToBeClickable(element){
try{
for(var i = 0; i <= 3000; i++){
var wait = await browser.wait(this.EC.elementToBeClickable(element), i);
if(wait == true){
break;
}else{
//this is where I want to fail
}
}
}catch(err){
//this is where I want to fail
await console.log(`WAIT TO BE CLICKABLE FAILED:\n${element.parentElementArrayFinder.locator_.value}\n\nError:\n${err}\n`);
}
};
this would help me a LOT debugging my script since I'm working on VSC, but I can not seem to find a way yet to make the test FAIL and thereby to CLOSE the browser at the first fail, I've seen protractor-fail-fast and protractor-bail-fast but it seems to be for the jasmine test cases not for function, I WOULD REALLY APPRECIATE any help please, protractor its driving me a bit nuts lol...
//method
const checkData = () = {
return new Promise((resolve)=>{
if(true){
// Success case
return resolve(true)
}
else{
// Fail case
return reject(false)
}
})
}
//Specfile
it('sample test',async ()=>{
Let data = await checkData();
expect(data).toEqual(true);
})
Based on resolved value test will pass or fail
If you function throws error you can just use the done function that jasmine provides
Example:
it('should do something', async done => {
try {
await weDoSomething();
} catch (e) {
done.fail(e); // mark test as failed
}
done(); // mark test as done
}, 1 * 60 * 1000);
async function weDoSomething() {
throw Error('failed function');
}
Did you try simply re-throwing the error in the catch? I believe that should cause the test to fail.
Comments:
You do not need to await a console.log as it is an synchronous operation.
broser.waits throw exceptions when the element is not found within the timeout period, it appears your for loop is not required at all.
This function will wait 3000ms for the element to be clickable and if that does not happen it will throw an exception which will be caught in the catch. It will log the message and then rethrow the error which will fail your test (assuming the error is not being caught and handled further up)
async WaitToBeClickable(element){
try {
await browser.wait(this.EC.elementToBeClickable(element), 3000);
} catch (err) {
console.log(`WAIT TO BE CLICKABLE FAILED:\n${element.parentElementArrayFinder.locator_.value}\n\nError:\n${err}\n`);
throw new Error(err);
}
};
I'm developing my first gnome-shell-extension currently. In the extension, I want to execute a simple shell command and use the output afterwards, for which I use Gio.Subprocess like it is used in this wiki: https://wiki.gnome.org/AndyHolmes/Sandbox/SpawningProcesses
Currently, I have an argument like this one with some parameters: "ProgramXYZ -a -bc" which I pass as the argument vector argv as ['ProgramXYZ','-a','-bc']. This case works fine.
So let's say I would like to call two programs and combine the output with your approach, like: "ProgramXYZ -a -bc && ProgramB". My current output is correct in a normal terminal, but I'm not sure how to pass it to the Gio.Subprocess. Something like ['ProgramXYZ','-a','-bc','&&','ProgramB'] does not work, is there a way to achieve that or do i have to make two seperate calls?
Sorry, I haven't managed to finish that page (that's why it's in my sandbox 😉).
Here is our Promise wrapper for running a subprocess:
function execCommand(argv, input = null, cancellable = null) {
let flags = Gio.SubprocessFlags.STDOUT_PIPE;
if (input !== null)
flags |= Gio.SubprocessFlags.STDIN_PIPE;
let proc = new Gio.Subprocess({
argv: argv,
flags: flags
});
proc.init(cancellable);
return new Promise((resolve, reject) => {
proc.communicate_utf8_async(input, cancellable, (proc, res) => {
try {
resolve(proc.communicate_utf8_finish(res)[1]);
} catch (e) {
reject(e);
}
});
});
}
Now you have two reasonable choices, since you have a nice wrapper.
I would prefer this option myself, because if I'm launching sequential processes I probably want to know which failed, what the error was and so on. I really wouldn't worry about extra overhead, since the second process only executes if the first succeeds, at which point the first will have been garbage collected.
async function dualCall() {
try {
let stdout1 = await execCommand(['ProgramXYZ', '-a', '-bc']);
let stdout2 = await execCommand(['ProgramB']);
} catch (e) {
logError(e);
}
}
On the other hand, there is nothing preventing you from launching a sub-shell if you really want to do shell stuff. Ultimately you're just offloading the same behaviour to a shell, though:
async function shellCall() {
try {
let stdout = await execCommand([
'/bin/sh',
'-c',
'ProgramXYZ -a -bc && ProgramB'
]);
} catch (e) {
logError(e);
}
}
I just realized all of my test code has a race condition.
My style pattern follows something like this:
const myFunc = (callback) => {
return somePromise().then((result) => {
return someOtherPromise();
}).then((result) => {
db.end(() => {
callback();
});
}).catch((err) => {
db.end(() => {
callback(err);
});
});
};
I'm testing with Jest. Test code looks something like this.
it('should work', (done) => {
// mock stuff
let callback = () => {
expect(...);
done();
};
myFunc(callback);
});
I have dozens of functions and tests following this pattern. The last test I wrote was giving me a Jest matcher error in my callback. After much confusion, I realized that the first callback execution is throwing the Jest failure, and the callback with the err parameter is being executed and failing before done() is called by the first callback execution.
I'm realizing this pattern might be absolutely terrible. I've managed to beat the race condition by having certain expect() calls in certain order, but that's no way to do this.
How can I remove the potential for the race condition here?
I'm open to completely changing the style I do this. I know my Javascript isn't particularly amazing, and the system is still pretty early in its development.
My colleague advised me that this is a good case to use async/await.
See a new version of the code under test:
const myFunc = async (callback) => {
let other_result;
try {
let result = await somePromise();
other_result = await someOtherPromise(result);
} catch (err) {
db.end(() => {
callback(err);
});
return;
}
db.end(() => {
callback(null, other_result);
});
};
I updated things a bit to make it seem more real-world-ish.
I know this makes myFunc return a promise, but that's okay for my use-case. By doing this, I'm ensuring that the callback is only executed once, preventing the Jest error from getting caught up elsewhere.
EDIT:
I'm realizing that this is the same as if I had moved the catch block to be before the final then block, I would have the same behavior :/
I have the following test:
it.only('validation should fail', function(done) {
var body = {
title: "dffdasfsdfsdafddfsadsa",
description: "Postman Description",
beginDate: now.add(3, 'd').format(),
endDate: now.add(4, 'd').format()
}
var rules = eventsValidation.eventCreationRules();
var valMessages = eventsValidation.eventCreationMessages();
indicative
.validateAll(rules, body, valMessages)
.then(function(data) {
console.log("SHOULD NOT GET HERE");
should.fail("should not get here");
done();
})
.catch(function(error) {
console.log("SHOULD GET HERE");
console.log(error);
});
done();
});
The test execution path is correct. When I have validating data, it goes to "SHOULD NOT GET HERE". The test is really to make sure it doesn't. And when I put in non validating data the code does go to "SHOULD GET HERE". So the validation rules work.
What I'm trying to do is make sure the test fails when when I have bad validation data and it validates. However when I run it as it is with good data it validates, runs the fail, but mocha still marks it as the passing. I want it to fail if the execution gets to "SHOULD NOT GET HERE".
I've tried throw new Error("fail"); as well with no luck. In both cases it actually seems to run the code in the .catch block as well.
Any suggestions?
I found solutions for this in a similar question. This question is written because those solutions don't seem to be working for me.
You can call assert.fail:
it("should return empty set of tags", function()
{
assert.fail("actual", "expected", "Error message");
});
Also, Mocha considers the test has failed if you call the done() function with a parameter.
For example:
it("should return empty set of tags", function(done)
{
done(new Error("Some error message here"));
});
Though the first one looks clearer to me.
In the ES2017 async/await world chai-as-promised is not needed as much. Although simple rejections are one place chai-as-promised remains a little neater to use.
A catch is required if you want to test the error in more detail.
it.only('validation should fail', async function(){
let body = { ... }
let rules = eventsValidation.eventCreationRules()
let valMessages = eventsValidation.eventCreationMessages()
try {
await indicative.validateAll(rules, body, valMessages)
} catch (error) {
expect(error).to.be.instanceOf(Error)
expect(error.message).to.match(/Oh no!/)
return
}
expect.fail(null, null, 'validateAll did not reject with an error')
// or throw new Error('validateAll did not reject with an error')
})
async/await requires Node.js 7.6+ or a compiler like Babel
Use chai-as-promised, with native Mocha promise handlers.
var chai = require('chai').use(require('chai-as-promised'));
var should = chai.should(); // This will enable .should for promise assertions
You no longer need done, simply return the promise.
// Remove `done` from the line below
it.only('validation should fail', function(/* done */) {
var body = {
title: "dffdasfsdfsdafddfsadsa",
description: "Postman Description",
beginDate: now.add(3, 'd').format(),
endDate: now.add(4, 'd').format()
}
var rules = eventsValidation.eventCreationRules();
var valMessages = eventsValidation.eventCreationMessages();
// Return the promise
return indicative
.validateAll(rules, body, valMessages)
.should.be.rejected; // The test will pass only if the promise is rejected
// Remove done, we no longer need it
// done();
});
This simple throw is working for me
describe('Lead', () => {
it('should create a new lead', async () => {
throw 'not implemented'
})
})