I'm writing a unit test framework. I noticed Mocha purports to handle exceptions in your test code gracefully, mapping the exception to the correct test in the log output. I'm attempting to do the same thing here. Let's ignore mapping the exception to the correct test for now (I'll def take a suggestion though, I'm stumped on how I'll do that for now)
I add all the test functions into the testmap, iterate over the keys in the testmap, and call each test's function one by one. I stall till the tests report back completed. I wrap around this a try-catch block to catch any exceptions that happen in any of the tests. It works - I can catch exceptions generated in the tests but even though I catch them, the program terminates. I do not want the program to terminate, and I don't think it's supposed to if you catch the exception... what gives?
Here is the try-catch in my library code
this.runtests = () => {
try {
Object.keys(this.testmap).forEach((test) => {
performance.mark(test)
this.testmap[test].testfunc();
});
this.stalltillgood(() => {
this.finallog();
});
}
catch (e) {
console.log('UNHANDLED EXCEPTION IN TEST');
console.log(e.stack);
}
};
Here is the client code which generates the exception - it is ENOENT - no such file or directory - hello.htm doesn't exist (on purpose)
expected('<fuck>&<fuck>');
testinfo('A Hello World Test', //the name of the test [MUST PASS TO ACTUAL AS WELL]
'This is the hello world test doc lol'); //the doc
comparator(cmp.equals); //you can use the pre built compare functions or your own
test(() => { //pass your test function to test
const file = fs.readFileSync('./hello.htm');
actual('A Hello World Test', file.toString());//make a call to actual in your test code
}); //to pass it your test result
//write more tests with the same sequence of commands
I think my problem is that this.runtests is the last method called, and after continuing on from my catch block, the program literally never has anything to output to me again, everything should be logged by then. The program just terminates after the catch block. I think I will have an extra var in test 'started' and just restart this.runtests, which will now check to see if a test has been started before trying to start it! Hooray! Still don't know how to map the exception to the proper test, maybe e.stack? Actually yeah that should be easy I guess lol.
I think my problem is that this.runtests is the last method called, and after continuing on from my catch block, the program literally never has anything to output to me again, everything should be logged by then. The program just terminates after the catch block. I think I will have an extra var in test 'started' and just restart this.runtests, which will now check to see if a test has been started before trying to start it! Hooray! Still don't know how to map the exception to the proper test, maybe e.stack? Actually yeah that should be easy I guess lol.
Related
So I have been developing some codes using AWS Lambda with NodeJS 6.10. Because of my lack of knowledge in integration testing (don't worry, the unit tests are done), I didn't test my code. Of course, I missed a bug that caused two sleepless nights. It keeps running even after I put in this
return workerCallback(err);
I thought it would stop the function from running other codes past the if clause because I returned it. Anyway, I was able to fix my issue by adding a return just after the asynchronous function
SQSService.deleteMessage
is called. The rest of the codes did not run and the lambda function ran and ended as expected.
Here are now the code that works as expected.
function myFoo(currentRequest,event, workCallback){
var current_ts = moment();
var req_ts = moment(currentRequest.request_timestamp);
if (req_ts.diff(current_ts, 'minutes') > 5) {
SQSService.deleteMessage(event.ReceiptHandle, function(err, data){
if (err) {
return workerCallback(err);
} else {
return workerCallback(null, "Request stale! Deleting from queue...");
}
}); //end of SQS Service
return; //This line... this line!
}
/* Codes below will execute because the code above is asynchronous
but it should not as the asynchronous code above is a terminator function
from a business logic point of view
*/
//more codes that will run should currentRequest.request_timestamp is 5 minutes past
}
Can someone please guide me on how to test this code or create a test that would at least prevent me from doing the same mistake again? I'd like to avoid these mistakes from happening again by testing. Thanks!
(I'm moving it to an answer so the comments thread doesn't fill up - and so I can type more).
The key is to get the proper grasp of async-ness in your code. myFoo seems to be asynchronous, so you need to decide whether all errors or failure modes should be handled as errors passed to its callback handler, or whether some types of error should return synchronous errors to the caller of myFoo itself. My general approach is, if any errors are going through the callback handler, to have them all go there - with the minor exception of certain types of bad-coding errors (e.g. passing in things of the wrong type, or passing in null for arguments that should always have variables) which I might throw Error() for. But if this kind of error (project_ref_no == null) is the kind of error that you should handle gracefully, then I'd probably pass it through to the error handler. The general idea is that, when you call myFoo, and it returns, all you know is that some work is going to get done at some point, but you don't know what is going to happen (and won't get a result in the response) - the response will come back later in the call to the callback handler).
But, more importantly, it's key to understand what code is being run immediately, and what code is in a callback handler. You got tripped up because you mentally imagines the internally generated callback handler (passed to SQSService.deleteMessage) was being run when you called myFoo.
As for testing strategy, I don't think there's a silver bullet to the issue of mistaking asynchronous calls (with callback handlers) with code that is run synchronously. You could sprinkle assertions or throw Error()'s all over the place (where you think code should never get to), but that'd make your code ridiculous.
Typescript helps with this a bit, because you can define a function return type, and your IDE should give you a warning if you've got code paths that don't return something of that type (something most/all? typed languages give you) - and that would help somewhat, but it won't catch all cases (e.g. functions that return void).
If you're new to javascript and/or javascript's asynchronous models, you might check out the following link:
https://medium.com/codebuddies/getting-to-know-asynchronous-javascript-callbacks-promises-and-async-await-17e0673281ee
According to wdio's documentation found here I should be allowed to handle an error using
browser.on('error', (err)=>{ console.log('do something') })
However, when I do this, nothing happens it isn't executed when the test fails, which from my understanding should have happened.
Am I doing this wrong? If so, how can I use the .on function to handle the test fails or errors my own way? I do know I can use try/catch blocks, but I would like something cleaner that is just called every time tests fail or there is an error.
Edit: as requested, added some code to show the error
browser.on('error', () => {
console.log('do something about the error')
})
it('should do something', function () {
const nonExistingElement = browser.getText('.idontexist')
assert.deepStrictEqual(nonExistingElement, 'Say something that isnt there')
})
What I would like to do is for the callback function of browser.on be executed whenever an error or failed test occur, in this case, I am trying to get the text of an element that doesn't exist, so naturally I will get the whole
element (".idontexist") still not existing after 500ms
But before that, as it happens I would also like to get the:
do something about the error
So far, the alternative to this, would be to put a try and catch block in my it block and do a browser.emit('error', 'Error'), which I would like to avoid so that I don't have to repeat it every single test.
I can share frustration with a lot of professionals about default protractor behavior on test failure - it just keeps running the tests and you have to wait until it finishes to fix the error.
I read the related posts and came across jasmine-bail-fast solution. Link to related post is provided here: Exit Protractor e2e test on fail?
However, this solution puts me on the other side of the pickle. I do NOT want to terminate the testing suite when it failed to close the confirmation message or ran into similar minor issues.
I would like to have the ability to control when to exit the script via exitonfailure() function or something similar. For instance, if had this block of code:
> browser.wait(function()
> return browser.isElementPresent(acceptBudgetButton);
> }, 30000, 'Error - Unable to save budget changes because Accept Budget button is not visible. Terminating test run.');
and put exitonfailure() after this block, I want my test run to exit immediately. However, I want to test keep on running if exitonfailure() is not there.
Is there a feasible way to achieve this goal and take control of my own destiny?
Thank you for your help!
You can handle the browser.wait() success and failure cases appropriately:
var EC = protractor.ExpectedConditions;
browser.wait(EC.presenceOf(acceptBudgetButton), 30000).then(
function () {
// success
},
function () {
// failure
console.log('Error - Unable to save budget changes because Accept Budget button is not visible. Terminating test run.');
jasmine.getEnv().bailFast();
}
});
Also, look into using fail() function which fails a test.
I'm writing a script in Nightwatch that tests a specific element on a page. It's possible that the script could be testing a URL in which the element is not present on the page, in which case I want the script to end the test without any failures being logged.
I cannot seem to find a way to abort the test early without invoking a failure, however. Is there any way to have a Nightwatch test abort on a pass?
Here's a part of the code I'm working with for reference:
//End test if pagination is not present
'Pagination Present' : function (browser) {
browser
.execute(function() {
return document.querySelectorAll("ul[class='pagination']").length;
},
function(count){
if (count.value == 0) {
browser.assert.equal(count.value, 0, "There is no pagination on this page.");
browser.end();
}
})
},
Invoking browser.end(); closes the browser, but it reopens immediately after and the tests continue. Every single case fails, since the pagination does not exist on the given page. I'd like to end the test immediately after browser.assert.equal passes. Is there any way to do so?
You can use try/catch.
I had the same issue with some tests and i've got it to skip that assertion like this: you try to check something, but if you don't find the element, instead of failing the test, i just display a message in the console and exit the whole test:
'Test product\'s UPSs' : function (browser) {
try {
browser.assert.elementPresent('#someElement');
}
catch(err) {
console.log('this product has no Special features! Skipping');
process.exit();
}
}
In case you have further tests that you know they wouldn't fail and you'd like to continue with them, just leave out the process.exit() function. While it might not be the safest way to do it, at least it gets the job done.
Is there any possible way to stop the automation testing? Actually i m testing my app with automation javascript. I have two test in my script. if the first test fail i dont want to continue my script..
MY code:
var target = UIATarget.localTarget();
var testname = "Test1";
UIALogger.logStart(testname);
target.frontMostApp().mainWindow().buttons()[0].tap();
UIALogger.logPass(testname);
target.delay(2);
var testname = "Test2";
UIALogger.logStart(testname);
target.frontMostApp().mainWindow().buttons()[1].tap();
target.frontMostApp().mainWindow().buttons()[0].tap();
UIALogger.logPass(testname);
Here if the test1 fails i have to say the script to stop the process.. please tell me any suggestion to this problem. Thanks...
You should really look at abstracting out your tests. You can use something like:
function test(description, steps) {
try{
UIALogger.logStart(description);
steps();
UIALogger.logPass("Test Passed");
} catch (exception) {
UIALogger.logError(exception.message);
UIATarget.localTarget().logElementTree();
UIALogger.logFail("Test Failed");
throw exception;
}
}
as a function you can wrap your test in. As you can see, it will start the test, complete the steps of your test (if possible). If it failed, the test will log the element tree (very handy) and then will throw the same error again, stopping the test from continuing, and giving information about the error you experienced.
I highly recommend using tuneup_js for running UIAutomation. I don't think you can stop the Test run if one test fails using Instruments GUI, but it will stop when using the tuneup_js library and running the tests from the command line