A question for the testers out there:
Consider that you're writing a test like this: One opens a modal, flips a toggle to "on", saves its state (which closes the modal), then opens it again to check that the state was saved successfully. You also have to check that the toggle, when flipped back to "off", saves successfully and maintains the toggle's "off" state.
Is it reasonable to write your tests so that they chain off the previous test? It feels painfully inefficient to not chain them, especially if your starting process involves logging in, navigating to some page, clicking some tab, then getting started on the stuff you actually want to test.
Although flipping to "on" and flipping to "off" are, strictly speaking, separately testable items, there's no need for you to test them in complete isolation - especially if it means that you have to reset data, re-do logins and navigation etc.
If you're confident that the two tests work together without one interfering with or polluting the other one, then just carry on with one single test.
Automated testers do need to be aware of the efficiency of their tests, and the amount of resources and time it takes to run the complete suite, because if they are too slow or expensive there will be pressure to cut-back.
Related
In the context of web-development many times users provide screenshots of their "invalid state".
I use React and I was wondering if we could debug the opposite way: starting from the current state:
User gives us screenshot of invalid UI state.
Via devtools we change props of components to match & find that specific permutation (let's imagine all components are stateless).
Feed the permutation to "a program" that shows us the things in our code that caused the props to change that way.
From a technical point of view I wonder whether this is even possible: can we go backwards in time from a specific state of a program?
This should be possible since the state is derived from a specific execution path of our code; so the opposite should also be possible? The reason I am interested is for debugging purposes but I also feel it would be an interesting exploratory excercise.
Are there any resources or tools that I can look into to find if this is possible?
EDIT: to make this a bit more clear I'm looking for a way to give the system as input a "state", and return a possible list of "path traces" in time that my code could execute to arrive at such state:
showHowWeCanArriveTo({
name: 'Bob',
showPopup: true,
colors: ['blue', 'green']
});
// Returns:
[
[setName('Bob'), setShowPopup(true), addColor('blue'), addColor('green')]
]
I've used Logrocket before in react for that exact same purpose. If you're using redux, they provide a middleware that logs the user state and its history and is accessible from their web.
You might be looking for time travel debugging, reverse debugging, or replay debugging. Basically, they work by recording what the program is doing as it executes and keeping enough logs that you can then go backwards.
Alternatively, it is possible in principle to use techniques from program analysis (program slicing, reverse program execution, etc.) to try to execute the program in reverse and find all execution paths that might lead to that state. However, there might be many such paths (the further back you go, the more there might be), and implementing such a thing from scratch is technically non-trivial.
I recently watched a number of talks from the AssertJS conference (which I highly recommend), among them #kentcdodds "Write Tests, Not Too Many, Mostly Integration." I've been working on an Angular project for over a year, have written some unit tests, and just started playing with Cypress, but I still feel this frustration around integration tests, and where to draw the lines. I'd really love to talk to some pro who does this day in and day out, but I don't know any where I work. Since I'm tired of not being able to figure this out, I thought I'd just ask the world here, cause you all are fantastic.
So in Angular (or React or Vue, etc), you have component code, and then you have the HTML template, and usually they interact in some way. The component code has functions in it that can be unit tested, and that part I'm ok with.
Where I haven't gotten things straight in my mind is, do you call it an integration test when you're testing how a component function changes the UI? If you're testing that kind of thing, should that be done just in E2E tests? Because Angular/Jasmine(or Jest) lets you do this kind of thing, referencing the UI:
const el = fixture.debugElement.queryAll(By.css('button'));
expect(el[0].nativeElement.textContent).toEqual('Submit')
But does that mean you should? And if you do, then do you not cover that in your E2E tests?
And regarding integration with things like services, how far do you go with integrating? If you mock the actual HTTP call, and just test that it would get called with the right functions, is that an integration test, or is it still a unit test?
To sum up, I intuitively know what I need to test to have confidence that things are working as they should, I'm just not sure how to discern when something requires all three kinds of tests or not.
I know this is getting long, but here's my example app:
There's a property called hasNoProducts that is set after a product is chosen and data is returned from the server (or not if there is none). If hasNoProducts is true, UI (through an *ngIf) shows that "Sorry" message. If false, then other selections become available. Depending on the product picked, those options change.
So I know I can write a unit test and mock the HTTP request so that I can test that hasNoProducts is set correctly. But then, I want to test that the message is displayed, or that the additional options are displayed. And if there is data, test that switching the product changes the data in the other lists that would subsequently show on screen. If I do that using Angular/Jasmine, is it an integration test since I'm "integrating" component and template? If not, then what would be an integration test?
I could keep asking questions, but I'll stop there in the hopes that someone has read this far and has some insight. Again, I've read tons of articles, watched tons of videos and done tutorials, but every time I sit down to apply to a real project, I get stuck on things like this, and I want to get past this! Thanks in advance.
What distinguishes unit-tests and integration-tests (and then subsystem-tests and system-tests) is the goal that you want to achieve with the tests.
The goal of unit-testing is to find those bugs in small pieces of code that can be found if these pieces of code are isolated. Note that this does not mean you truly must isolate the code, but it means your focus is the isolated code. In unit-testing, mocking is very common, since it allows to stimulate scenarios that otherwise are hard to test, or it speeds up build and execution times etc., but mocking is not mandatory: For example, you would not mock calls to a mathematical sin() function from the standard library, because the sin() function does not keep you from reaching your testing goals. But, leaving the sin() function in does not turn these tests into integration tests. Strictly speaking, you could even have unit-tests where some real network accesses take place (if you are too lazy to mock the network access), but due to the non-determinism, delays etc. these unit tests would be slow and unreliable, which means they would simply not be well suited to specifically find the bugs in the isolated code. That's why everybody says that "if there is some real network access, it is not a unit-test", which is not formally but practically correct.
Since in unit-testing you intentionally only focus on the isolated code, you will not find bugs that are due to misunderstandings about interactions with other components. If you mock some depended-on-component, then you implement these mocks based on your understanding of how the other component behaves. If your understanding is wrong, your mock implementations will reflect your wrong understanding, and your unit-tests will succeed, although in the integrated system things will break. That is not a flaw of unit-testing, but simply the reason why there are other test levels like integration testing. In other words, even if you do unit-testing perfectly, there will unavoidably remain some bugs that unit-testing is not even intending to find.
Now, what are integration tests then? They are defined by the goal to find bugs in the interactions between (already tested) components. Such bugs can, for example, be due to mutual misconceptions of the developers of the components about how an interface is meant to work. For example, in the case of a library component B that is used from A: Does A call functions from the right component B (rather than from C), do the calls happen while B is already in a proper state (B might not be initialized yet or in error state), do the calls happen in the proper order, are the arguments provided in the correct order and have values in the expected form (e.g. zero based index vs. one based index?, null allowed?), are the return values provided in the expected form (returned error code vs. exception) and have values in the expected form? This is for one integration scenario - there are many others, like, components exchanging data via files (binary or text? which end-of-line marker: unix, dos, ...?, ...).
There are many possible interaction errors. To find them, in integration testing you integrate the real components (the real A and the real B, no mocks, but possibly mocks for other components) and stimulate them such that the different interactions actually take place - ideally in all interesting ways, like, trying to force some boundary cases in the interaction (exchanged file is empty, ...). Again, just the fact that the test operates on a software where some components are integrated does not make it an integration test: Only if the test is specifically designed to initiate interactions such that bugs in these interactions become apparent, then it is an integration test.
Subsystem tests (which are the next level) then focus, again, on the remaining bugs, that is, those bugs which neither unit-testing nor integration testing intend to find. Examples are requirements on the component C that were not considered when C was decomposed into A and B, or, if C is built using some outdated version of A where some bug was still in. However, when climbing up from unit-testing via integration testing to subsystem testing and above, it is a challenge to stay focused: Only to have tests for bugs that could not have been found before, and not to, say, repeat unit-tests on subsystem level.
Although I have been writing unit tests for 20-odd years, I am new to Gherkin, and haven been given the task of implementing a story for a .feature file that reduces to something like this:
Scenario: a
Given that the app is open
When I open a certain dialog
Then it has a thing somewhere
Scenario: b
Given that the dialog from 'a' is open...
# Imagine here a long chain of scenarios, each depending on the previous
Scenario: n
Given that the previous 'n' steps have all completed....
That is, a long, long chain of scenarios, each depending on the state of the system as configured by its predecessor.
This doesn't feel right to someone used to unit testing -- but these scenarios are not going to be split and run separately.
What's the best practice here?
Should I rewrite into one very long scenario?
I am already using a 'page object' to keep most of my code out of the step definitions -- should I be coding the steps as single calls, that can be re-used in the later scenarios?
I'm running Cucumber in Javascript.
First things first, Warning:
For the majority of tests (and by majority I mean 99.9% of the time), you should not carry on from the previous scenario, because of the fact that if one scenario fails in your feature, more will crumble because you attempted to string them together.
And on to my answer:
Depending on whether you are trying to do a set up for all of your scenarios after (within the same feature), or whether you want to reuse that first scenario multiple times (in separate features), you could do one of 2 things.
Make the first scenario a background
Make the first scenario into a step definition, for use in multiple feature files
For the First:
Background:
Given that the app is open
When I open a certain dialog
Then it has a thing somewhere
Scenario: a
Given that the dialog from 'a' is open...
Just remember that when you use it as a background, it will be used for all the following scenarios within that feature.
For the Second:
Scenario: a
Given that the app is open
When I open a certain dialog
Then it has a thing somewhere
Scenario: b
Given I have opened the dialogue from a
And the '<DialogFromA>' dialog is open...
I would ask myself, what is the actual behaviour behind all the steps?
Then I would implement that as the wanted behaviour and push the order between steps down the stack. Probably using one or many helper classes. There is nothing saying that you can force the order of the scenarios without introducing some hack to force them together.
Remember that BDD and Cucumber is all about human readable communication. The dependencies you are asking for should, in my opinion, be implemented in the support code Gherkin triggers.
I am building a Meteor application that is using a mongo database.
I have a collection that could potentially have 1000s of documents that need to be updated at different times.
Do I run setTimeouts on creation or a cron job that runs every second and loops over every document?
What are the pros and cons of doing each?
To put this into context:
I am building an online tournament system. I can have 100s of tournaments running which means I could have 1000s of matches.
Each match needs to absolutely end at a specific time, and can end earlier under a condition.
Using an OS-level cron job won't work because you can only check with a 60-second resolution. So by "cron job", I think you mean a single setTimeout (or synced-cron). Here are some thoughts:
Single setTimeout
strategy: Every second wake up and check a large number of matches, updating those which are complete. If you have multiple servers, you can prevent all but one of them from doing the check via synced-cron.
The advantage of this strategy is that it's straightforward to implement. The disadvantages are:
You may end up doing a lot of unnecessary database reads.
You have to be extremely careful that your processing time does not exceed the length of the period between checks (one second).
I'd recommend this strategy if you are confident that the runtime can be controlled. For example, if you can index your matches on an endTime so only a few matches need to be checked in each cycle.
Multiple setTimeouts
strategy: Add a setTimeout for each match on creation or when the sever starts. As each timeout expires, update the corresponding match.
The advantage of this strategy is that it potentially removes a considerable amount of unnecessary database traffic. The disadvantages are:
It may be a little more tricky to implement. E.g. you have to consider what happens on a server restart.
The naive implementation doesn't scale past a single server (see 1).
I'd recommend this strategy if you think you will use a single server for the foreseeable future.
Those are the trade-offs which occurred to me given the choices you presented. A more robust solution would probably involve technology outside of the meteor/mongo stack. For example, storing match times in redis and then listening for keyspace notifications.
This is all a matter of preference, to be honest with you.
I'm a big fan of writing small, independent programs, that each do one thing, and do it well. If you're also like this, it's probably better to write separate programs to run periodically via cron.
This way you get guaranteed OS-controlled precision for the time, and small, simple programs that are easy to debug outside the context of your webapp.
This is just a preference though.
In a web app which I'm building, I have two loosely related bits of code running in two separate timers every one second.
I'm looking to optimize the Javascript, is it worth merging these two timers into one or is that just over the top?
Realistically, am I going to increase any performance (considering that we don't know what sort a system a visitor is running ) by merging two 1 second intervals into one 1 second interval?
As I understand it, JavaScript is single threaded so the more things happening, the more these stack up and block other things from happening (timers especially). I just don't know whether one measly timer running every second is an issue at all.
The reason for keeping the two timers separate would purely be code readability, which is fine on the server side where you control the hardware but I don't know what sort of browser or hardware my visitors will be running.
Thanks.
In terms of the overall number of operations that can be completed, no, there isn't going to be a measurable difference. It is possible for there to be a perceived performance advantage in keeping multiple timers, however. The more code you have running synchronously in a single timer iteration, the longer all DOM updates and certain types of user interactions are "halted". By splitting these up into multiple timers, you allow other updates to take place in between timer iterations, and therefore the user gets a "smoother" experience.
Odds are in this case there won't even be a difference in perceived performance either, though, so I'd do it whichever way makes the code organization simpler.
If performance really is an issue you could just create 1 timer, and for example use that to call both functions:
function update()
{
A(); //Do your first task
B(); //Do the second
setTimeout("update()", 1000);
}
update();
However, how sure are you that the bottleneck is within this timer? Try to measure first, and dont optimise the wrong parts of your application.
I would bet that you'd increase performance by eliminating clock handling at the JS level. You certainly won't degrade performance and, with just one timer running, I'd think that you'd enhance code maintainability, if not readability. In the app I'm working on right now, I have one timer running to handle three tasks: a special kind of scrolling, changing the background image of perhaps 300 cells, and checking to see if it's time to refresh the page and issuing an AJAX request if so. That timer is running with a 1/10-sec interval and things are tight, for sure, but the code gets through all of those jobs, once in a while with one clock tick coming on top of the previous.
So I doubt you'll have any trouble with a 1-sec interval and just one tick handler.