How to Write Effective Defect Reports
No matter how much test automation you have built-in to your overall test strategy, there will always be some amount of manual testing for things like edge cases, one-offs, visual acceptance tests, etc…You also want customers to have the ability to log defects they experience when using the system under test. In this blog post we'll be diving into defect reporting and defect report templates, along with a baseline recommendation for the latter.
We defined the differences between bugs, defects, errors, and failures in a previous blog post: “A defect is deviation from requirements that causes a component or system to fail; the flaw itself and NOT the result. A bug is really a slightly narrower definition of a defect, as its specifically a software fault that causes a component or system to fail. An error is a human action that causes a fault or mistake, like defects. And finally, a failure is the inability of a component or system to perform an expected function as specified in the requirements; the result of a defect.”
The defect report should be part of a larger defect management process where ultimately the defect is verified as a genuine issue that hasn’t previously been reported, and then its eventually closed/resolved. There is no one size-fits-all for process. For example, you may want to have a slightly different style of of defect reporting for customers than the one your in-house testing team uses - the latter might include a “testing phase” field and/or an “issue open/closed” toggle. Regardless of the specific defect management process flow, no defect report should ever marinate too long in your system or, even worse, disappear into the digital netherworlds!
The primary purposes of defect reporting and tracking include:
The end goal of course is to continually improve the system under test: The user journey through the system should be smooth and productive, and the experience should only improve over time due to timely fixes and feature enhancements.
The more informative, detailed, and clear the defect report then the less time the testers, developers, or whoever is charged with acting on it have to spend deciphering and trying to understand the defect in question (see good and bad examples below). Very often a bug that can’t be reproduced will not be fixed. Also, the focus should be on reporting the defect and related details - and not in any way a dissertation on how the developers should have built the system under test (aka "smug report").
This might be a controversial opinion...but the more concise the defect report template, then the more likely it will actually be filled out in a timely manner. Everyone is busy and often doing multiple jobs these days, and so it’s easy for someone to look at a dauntingly gargantuan defect template and procrastinate or even let it slip through the cracks. We want fast feedback on our system defects, and so any stakeholder who comes across a problem should submit their defect report as soon as possible. For those folks who love to be meticulous and/or overshare – and both are a good thing when it comes to defect reports – they can simply add more information in the “description” field.
Defect Report Template:
Defect ID: Unique identifier associated with each defect; your defect management system should automatically generate this
Date: Your defect management system can populate the date, but it should allow the reporter to manually change this if the defect occurred a day or two earlier and they are reporting it late
Severity: There should be some form of scale from minor issue to a reverse big bang where the Universe collapses on itself
Reporter Name: This should be populated automatically, linked to defect management system login
Platform/Environment: Testers can get so wrapped up in the weeds of how the defect occurred, that they can forget key environment information; hardware information beyond brand and/or CPU maker can get overly in-depth and so I would leave it up to the reporter as to how much of that to include
Any Additional Dependencies: Worth noting if additional variables are involved, like if applications or SDKs related to the issue were installed
Build Version: The version number is absolutely critical for reproducing and fixing the defect
Description: Summarize everything related to the defect, and add any relevant information not covered in other parts of the defect template
Expected Result: This should be kept strictly to what is expected by the system within the boundaries of the user requirements, and NOT a lengthy manifesto on what the reporter would prefer the developers have done instead (that's a Smug Report)
Actual Result: The reporter should be very specific to note not just the defect, but anything else unexpected or weird
Steps to Reproduce: Think of this as the “defect reproduction recipe” for anyone reading the report
Artifact Attachment(s): This includes screenshots, log files, and anything that could be used for reproducing, debugging, and testing; consider making this field prominent so a defect reporter in a hurry doesn’t miss it
Defect ID: DFT82321339
Severity: Level III
Reporter Name: Otto Mation
Platform/Environment: Chrome v.84.0.4147 in Windows 10 v.2004 on a newer Dell w/ Intel CPU
Any Additional Dependencies: None
Build Version: PomeloShop Version 3.2
Description: This issue was found when doing exploratory testing on the recently revamped customer portal; all portal login test cases will eventually be automated tests. Logging in with valid credentials to the Pomelo Shop login portal at “http://www.pomeloshop.com/index.php?route=account/login” results in an error message rather than the welcome screen. Clicking OK on the message refreshes the login page. I recall we experienced a similar error in version 2.2 when running the Safari Browser; worth checking previous defect records for similarities when debugging.
Expected Result: The user should be taken to the Pomelo Shop exotic fruit store returning customer welcome screen
Actual Result: Error message pops up stating “Your login information was found, but is somehow corrupted”
Steps to Reproduce:
- Enter valid username “email@example.com”
- Enter valid password “OttoPass123”
- Click Login button
Artifact Attachment(s): Screenshot
Defect ID: DFT82321339
Date: 8/4/20 (late reporting)
Severity: Kinda Bad
Reporter Name: Otto Mation
Platform/Environment: My favorite browser in Windows
Any Additional Dependencies: Installed some stuff from sketchy gambling sites
Build Version: The earlier PomeloShop one
Description: Login no work
Expected Result: The current Pomelo Shop fruit store welcome screen is boring; I want a fancy one with dancing exotic fruit mascots that shimmy and twist around the user’s screen
Actual Result: Some error popped up over top of the current boring welcome screen
Steps to Reproduce:
- Put stuff into login boxes
- Click button
Artifact Attachment(s): Nope
I don’t profess that the above template is perfect for all systems to be tested, types of users doing the reporting, or defect management processes, but I think at the very least it could be used as a solid starting point to expand upon. How does this template compare to what you use to report defects? Drop us a line and let us know!
This post was written by:
Technical Pre-Sales Consultant
James has been working in software pre-sales and implementation since 2000, and has more recently settled into focusing on technical pre-sales. He takes care of our hands-on demonstrations, and eagerly awaits your request to see our Cycle test automation software in action. Drop him a line at: james.prior[at]tryonsolutions[dot]com.
Global Engineering Group Upgrades WMS with Tryon Solutions & CycleOur customer is a global, high-technology engineering group founded in 1862. The company is a leader in tools and tooling systems for metal cutting; equipment, tools and services for the mining and...
Prominent Food and Beverage Customer Performs Regression and End-to-End Testing with CycleOur well-known food and beverage industry customer founded in 1897, has an extensive portfolio of consumer food and beverage brands. The company reported more than $7 billion in...
Essential Checklist for Deciding Which Tests to Automate To automate or not to automate?That is the question a lot of test professionals ask themselves- when deciding what tests should be run by software and which ones are best performed by people?To help, we've come...
What We Covered: High level explanation of Cycle as a behavior-driven test automation solution Overview of advantages including collaboration, lower maintenance, streamlining, and how Cycle is used to mitigate deployment risk Overview of Cycle interface, syntax, key...
The retail landscape is transforming--do you have the people and processes in place to excel? In this whitepaper, we outline the 5 ways to succeed in this consumer-driven landscape.