What Makes a Good Test Case?
We get distracted and are often hurried when designing test cases, as everything is on a tightly project-managed deadline these days…but are they good? What makes a “good” test case? Let’s start by level setting on a definition of a test case…Scouring the trusty Internets® for a universal definition of a “test case” is a futile exercise, and there is of course no accepted industry standard definition because that would make things too easy. We shall hereby define a test case as a set of conditions, variables, and/or actions that are performed on a system under test in order to validate that it meets requirements and verify that it functions correctly. Feel free to notify the Commandant of the Internets® as well as the Omnipotent Overseer of the software testing industry that we have decreed this as the official definition.
Good test cases have…
A clear objective with refined scope
What specifically is the intent and scope of the test? Is this a white or black box test, and is the purpose regression or performance? When determining the test objective, start at a high-level considering user context and then work down to thinking at a granular functional level. For example, are we only testing that a single login button was clicked or are we testing that an existing user can login to the website? There is certainly overlap between the two, but both are not necessarily the same test. Don’t hesitate to bring in more of your team at this stage to get their input; consider getting everyone together for an exercise like example mapping. If you botch the objective, which is really the overall point of the test, then all the work related to that test case that comes afterwards is a waste of time. So, you know, try not to do that.
Obvious and meaningful pass/fail verifications
What constitutes a “pass” and “failure”, and how are both determined? Each should be clearly defined as specifically as possible. Using the login example mentioned above, are we satisfied that an html “screen read” verification that sees “User Logged In” is designated a “pass”? Or, should our test case check a user log file on the web login server? Here is a more higher-level (though admittedly morbid) example: Let’s say we want to test whether the guy in the graphic below plunged to his doom or successfully makes the jump to the other side. We can check the target ledge to see if he is there, but what if he isn’t? Is the guy not being there a verification that he plunged to his doom? Something we didn’t prepare for could have happened, like a giant pterodactyl might have snatched him up mid-leap or maybe our hero spontaneously combusted. The only way to know that he plunged to his doom is to check the canyon floor for his crumpled body. The test case is only as accurate as what conditions you are verifying or validating.
Clear and concise documentation
Create a standardized template for your test cases recording things like a unique ID number, description, any pre-conditions, related datasets, and expected results; this is especially important for manual testing and/or if your test scripts are written in low-level code or are otherwise hard to read. If you have a test automation solution based on a business readable scripting language, then the readable scripts could potentially double as a test case template and related process documents.
Traceability to requirements
All test cases for a system under test should be traceable to the business requirements. We want to ensure that we have full test coverage of requirements and change requests while also ensuring that we are not wasting our time testing irrelevant components. Consider using something like a requirements traceability matrix to help with this effort.
Test cases will probably change over time as the system under test evolves throughout its life, but we certainly want to write test cases that will be reusable for as long as possible. Write the test case to be modular and easily maintainable. Let’s say the login example test case will be automated, if I know the button image will be changing in future versions then I should consider interacting with the web element instead of the image in order to make the test more maintainable and in-turn, more likely to be reused.
Independence from other test cases while testing one thing
A single test case shouldn’t depend on other test cases for execution. Ideally, to create larger end-to-end tests you should be able to combine your independent, modular test cases into test suites for sequential or parallel execution.
Permutations taken into account by the test case designer
Is it necessary to test every permutation or just the critical path? What about negative testing possibilities? In our web login example, we want to test valid username and password login combinations - but do we also want to check for the possibility of various kinds of invalid login credentials? What if users enter weird characters into the username and password fields? It might be ok to only test the critical path, especially if a tight deadline is looming, but you should at least have the debate where to draw the line with your testing.
This post was written by:
Technical Pre-Sales Consultant
James has been working in software pre-sales and implementation since 2000, and has more recently settled into focusing on technical pre-sales. He takes care of our hands-on demonstrations, and eagerly awaits your request to see our Cycle test automation software in action. Drop him a line at: james.prior[at]tryonsolutions[dot]com.
Vice President of Engineering
Evan Edwards is the Vice President of Engineering for Tryon Solutions and oversees the development of Cycle, our behavior-driven test automation solution that allows all personnel to join the testing fold and helps implementation teams to deploy with confidence. He ensures that Cycle grows in response to industry needs while remaining stable, streamlined, and easy-to-use.