I'm stuck with an issue with a cucumber test suite and I can't think of any way of debugging it.
We have a pretty sizeable suit of cucumber features, all of them pass on the development machines. The problem is a couple of scenarios are failing when we run the whole cucumber suite on our ci server, and running them individually makes them pass and fail (apparently) randomly when the scenario tries to fill a form (that apparently isn't on the page). Because of the random failures I thought it was a timing problem with an ajax request, but that doesn't seem to be the case, because adding really big sleep (tried everything from 1 to 60 secs) doesn't change anything. This scenario is even more fun because there's another 3 scenearios running the same steps that are failing on the first one in the same order, and these pass except if I delete the first scenario, in which case the first one to run those steps is the one that fails.
Is there any tricks to debug this sort of weirdness on cucumber? features (keep in mind these scenarios always pass on the dev machines, the problem is in the ci server).
Thanks!
I've also had the opportunity to debug intermittent test failures that only reproduce on CI. In my experience the problem always boiled down to few basic causes:
Race conditions in the front-end. For example, enabling a form for input before an xhr callback adds default values.
"Optimistic" writes from the front-end. Sometimes a front-end engineer makes an action involving a PUT/POST request more responsive by ignoring the result. In this case, there's no way to get Cucumber to wait until the request has completed, so a test against the state change in the database will have a race with the application.
Requests to resources that aren't available in the test fixture. For example, requests to 3rd party APIs might be blocked from CI. Sometimes URLs are not constructed correctly in the test environment, particularly when they are built "by hand", instead of using Rails helpers.
Intermittent Cucumber failures are always challenging to debug. Don't give up! It's worth the effort to figure out how to build a testable, race-free front-end. You can use capybara-webkit to debug the CI-only failures. Get the javascript console output to printed out on CI, and then you can add prints to your javascript to trace its state up to the point of test failure. You can also hack capybara-webkit to print out information about requests made by the front-end. Here is an example: https://github.com/joshuanapoli/capybara-webkit/commit/96b645073f7b099196c5d3da4653606e98a453e4
Related
I've written some E2E tests in cypress for a react application.
My team plans to put these tests in the CI/CD pipeline. The problem is, that the react app checks the login on every URL visit, logs in, and then continues with the E2E test.
In every "it" test, I visit the URL and have a wait of 1000ms implemented to let the page load properly. The problem is, that there are a lot of tests that make this testing really slow. One complete test group takes around 4000-5000ms and there would be more than 10-20 test groups. This would become really slow during the CI/CD.
Another problem is, that a lot of these tests implement the typing using the .type() function. It is really slow when we use this. Is there any workaround for this?
The last problem that I notice sometimes even when the elements have been rendered, the tests sometimes fail saying that the element was not found or was detached from the DOM, but when looking at the web page at that moment, I can clearly see the element. And re-running the tests, it passes. It becomes very uncertain and these tests also fail sometimes in the headless mode (which will be used in CI/CD I assume). Any comments on this?
Any suggestions/opinions on cypress + react in CI/CD?
to run the tests faster - use: (it can help to solve problem #2)
https://www.npmjs.com/package/cypress-parallel
We have built a webapp using Meteor.JS and we would like to find out
how will it perform in real production mode when thousands/millions of users can log request/response.
How efficient will be the framework when it comes to volume and response time.
We would like to know if there is any tools or best practices which we can use.
Thank you for your time in advance.
What you need to do is look at load testing your app. There are many ways to do this, but the premise is that you script what a user interaction consists of and you run that script multiple times, ramping up the volume until you get to the concurrency numbers you are looking for.
As mentioned, there are LOTS of tools to do this. Some I have personally used:
Loadrunner
Apache jMeter
Rational Performance Tester
Simple shell scripts with curl
There is a list on Wikipedia that would give you a good start on what other tools may be available.
The next point I'd like to make is that load testing is not just ramping up the concurrent users until the system breaks. A good load test will attempt to mimic the type of traffic you see or expect to see. To say it another way, two users do not do the exact same thing each time they hit your system, and not all requests to your server produce the same load on your system. Your users may even use the system differently at different times of the day.
The ultimate goal would be to have several "types" of users and transactions scripted, and have your load tool of choice weight the requests such that they match the expected percentages for each type. This way you can better see how your application will perform over time with loads that match what you really expect to see.
Finally, you need to consider your infrastructure. Typically, in the Enterprise environments I have worked in, we try to make our QA environment as close to Production as possible. We would have the exact same server hardware, exact same configuration (both physical and software), and attempt everything possible to make it mirror. Where we sometimes deviate is the size. For instance, I worked in on environment where we had 4 app servers in Prod, each with 4 nodes, for a total of 16 nodes in a cluster. In QA, we had the same bare metal hardware, but everything was cut in half, so we had 2 physical servers, each with 4 nodes totally 8 nodes in the QA cluster. When we tested for load, we would then half the expected total, so if we expected 200 users concurrently in Prod, a successful QA test would be 100 users concurrently. We would also copy and obfuscate data from the Production databases to the QA databases to make sure we are operating with the same sized data.
The process for load testing is this:
Grab a "baseline" result
Make whatever changes you need to make
Rerun the EXACT same tests as #1, only adding any new functionality to your tests
The baseline is important when making changes over time to ensure that your change doesn't effect performance. I can't tell you how many times I have seen SQL statements that ran perfectly fine in Dev that completely took down a system because of differences in the amount of data.
I've written some functional tests for the Intern, which supposedly should work on SauceLabs, BrowserStack, TestingBot, or my own Selenium grid.
The same code doesn't seem to work on all services though. I initially got the functional tests working on SauceLabs, so I'm using that service as my "base", so to speak.
On BrowserStack, the tests appeared to fail because the commands were executing too quickly. For example, I am using .pressKeys('TEST\nIN\nPROGRESS\n'), where \n is supposed to execute javascript on the page to turn the previous text into a tag (like the SO tags for this question: [intern] [javascript] [testing]).
That command should result in the following:
[TEST] [IN] [PROGRESS]
but instead results in
[TESTIN] [PROGRESS]
causing my assertions to fail. Changing the pressKeys command to
.pressKeys('TEST\n')
.sleep(500)
.pressKeys('IN\n')
.sleep(500)
.pressKeys('PROGRESS\n')
did not solve the issue. The test would pass / fail inconsistently, with the tags sometimes coming out as [TEST] [IN] [PROGRESS], and sometimes as [TESTIN] [PROGRESS].
Another example is that it wouldn't always wait for the next page to load when I .click() on a link, even with a .sleep() command after.
With regards to TestingBot, the application flat-out failed to upload files, and I could not for the life of me figure out how to enable the file_watcher service required to do so. They have a file upload example here, but I don't know how to configure the Intern to do this for me.
Isn't the Intern supposed to take care of these differences in the cloud providers for the tests?
Is there some standardized way of writing my tests in the Intern so that I can change my cloud testing provider without changing the tests themselves?
It should be possible to run the same test suite against any cloud-hosted Selenium providers and have them execute successfully, but there are some things you must do:
You need to make sure you’ve correctly configured providers so they all run the same version of Selenium. There is no standard for this; each provider uses a different key to decide which Selenium version to run. Check each provider’s documentation for the correct key to use.
You need to write tests that don’t have race conditions. What you’re describing here sounds like a classic race condition where you are performing some action that completes asynchronously, and so only happens in environments that execute operations within a certain period of time. Modifying this specific test so it has a find timeout and then tries to find the element you expect to be generated when the return key is hit should be a good solution, since this will allow you to wait as long as necessary without making your test slow.
Unfortunately, even with this advice, all of the cloud-hosted providers for Web browser testing are garbage and mess with things in a way that randomly causes tests to break. BrowserStack is the best by far at avoiding this, but even they do things to break tests from time to time that work perfectly well in a locally hosted Selenium installation.
For file uploads, Intern will automatically upload files if has detected that the remote provider supports it, and you type a valid path to a file on the server where intern-runner is running. You can check whether the server supports uploads by looking at this.remote.session.capabilities.remoteFiles. Feature detection must be turned on for this to work, and you should run Intern 3.0.6 or newer if you are trying to upload files to a Selenium server on the same machine as intern-runner.
I want to do a smoke test in order to test the connection between my web app and the server itself. Does Someone know how to do it? In addition I want to do an acceptance tests to test my whole application. Which tool do you recommend?
My technology stack is: backbone and require.js and jquery mobile and jasmine for BDD test.
Regards
When doing BDD you should always mock the collaborators. The tests should run quickly and not depend on any external resources such as servers, APIs, databases etc.
The way you would want to make in f.e. Jasmine is to declare a spy that pretends to be the server. You then move on to defining what would be the response of the spy in a particular scenario or example.
This is the best aproach if you want your application to be environment undependent. Which is very needed when running Jenkins jobs - building a whole infrastructure around the job would be hard to reproduce.
Make spy/mock objects that represent the server and in your specs define how the external sources behave - this way you can focus on what behavior your application delivers under specified circumstances.
This isn't a complete answer, but one tool we've been using for our very similar stack is mockJSON. It's a jQuery plugin that does a nice job both:
intercepting calls to a URL and instead sending back mock data and
making it easy to generate random mock data based on templates.
The best part is that it's entirely client side, so you don't need to set up anything external to get decent tests. It won't test the actual network connection to your server, but it can do a very good job validating that type of data your server would be kicking back. FWIW, we use Mocha as our test framework and haven't had any trouble getting this to integrate with our BDD work.
The original mockJSON repo is still pretty good, though it hasn't been updated in a little while. My colleagues and I have been trying to keep it going with patches and features in my own fork.
I found a blog post where the author explain how to use capybara, cucumber and selenium outside a rails application and therefore can be use to test a javascript app. Here are the link: http://testerstories.com/?p=48
I am new to Node.js and am currently questioning its reliability.
Based on what I've seen so far, there seems to be a major flaw: any uncaught error/exceptions crashes the server. Sure, you can try to bullet-proof your code or put try/catch in key areas, but there will almost always be bugs that slip through the crack. And it seems dangerous if one problematic request could affect all other requests. There are 2 workarounds that I found:
Use daemon or module like forever to automatically restart the server when it crashes. The thing I don't like about this is that the server is still down for a second or two (for a large site, that could be hundreds (of thousands?) of request).
Catch uncaught exceptions using process.on('uncaughtException'). The problem with this approach (as far as I know) is that there is no way to get a reference to the request that causes the exception. So that particular request is left hanging (user sees loading indicator until timeout). But at least in this case, other non-problematic requests can still be handled.
Can any Node.js veteran pitch in?
For automatic restarts and load-balancing, I'd suggest you check out Learnboost's up balancer.
It allows you to reload a worker behind the load-balancer without dropping any requests. It stops directing new requests towards the worker, but for existing requests that are already being served, it provides workerTimeout grace period to wait for requests to finish before truly shutting down the process.
You might adapt this strategy to be also triggered by the uncaughtException event.
You have got full control of the base process, and that is a feature.
If you compare Node to an Apache/PHP setup the latter is really just equivalent to having a simple Node server that sends each incoming request to it's own process which is terminated after the request has been handled.
You can make that setup in Node if you wish, and in many cases something like that is probably a good idea. The great thing about Node is that you can break this pattern, you could for instance have the main process or another permanent process do session handling before a request is passed to it's handler.
Node is a very flexible tool, that is good if you need this flexibility, but it takes some skill to handle.
Exceptions don't crash the server, they raise exceptions.
Errors in node.js that bring down the entire process are a different story.
Your best bet (which you should do with any technology), is just test it out with your application as soon as possible to see if it fits.
An uncaught exception will, if not caught, crash the server. Something like calling a misspelled function. I use process.on('uncaughtException') to capture such exceptions. If you use this then, yes, the error sent to process.on('uncaughtException') is less informative.
I usually include a module like nomnom to allow for command-line flags. I include one called --exceptions which, when set, bypasses process.on('uncaughtException'). Basically, if I see that uncaught exceptions are happening then I, in development, start up the app with --exceptions so that when that error is raised it will not be captured, which causes Node to spit out the stack trace and then die. This tells you what line it happened on, and in what file.
Capturing the exceptions is one way to deal with it. But, like you said, that means that if an error happens it may result in users not receiving responses, et cetera. I would actually recommend letting the error crash the server. (I use process.on('uncaughtException') in apps, not webservers). And using forever. The fact is that it is likely better for the webserver to crash and then expose what you need to fix.
Let's say you used PHP instead of Node. PHP does not abruptly crash the server (since it doesn't really serve). It spits out really ugly errors. Sure, it doesn't result in a whole server going down and then having to come back up. Nobody wants their clients to have any downtime. But it also means that a problem will persist and will be less noticeable. We've all seen sites that have said errors, and they don't get patched very fast. If such a bug were to take everything down for one small blip (which honestly its not all that bad in the larger picture) then it would surely call attention to itself. You would see it happen and would track that bug down.
The fact is that bugs will exist in any system, independent of language or platform. And it is arguably better for them to be fatal in order for you to know they happened. And over time it causes you to become more aware of how these error occur. I don't know about you, but I know a lot of PHP devs who make the same common mistakes time after time.