How to Write Effective Defect Reports

No matter how much test automation you have built-in to your overall test strategy, there will always be some amount of manual testing for things like edge cases, one-offs, visual acceptance tests, etc…You also want customers to have the ability to log defects they experience when using the system under test.  In this blog post we'll be diving into defect reporting and defect report templates, along with a baseline recommendation for the latter.

We defined the differences between bugs, defects, errors, and failures in a previous blog post: “A defect is deviation from requirements that causes a component or system to fail; the flaw itself and NOT the result.  A bug is really a slightly narrower definition of a defect, as its specifically a software fault that causes a component or system to fail.  An error is a human action that causes a fault or mistake, like defects.  And finally, a failure is the inability of a component or system to perform an expected function as specified in the requirements; the result of a defect.”

The defect report should be part of a larger defect management process where ultimately the defect is verified as a genuine issue that hasn’t previously been reported, and then its eventually closed/resolved.  There is no one size-fits-all for process.  For example, you may want to have a slightly different style of of defect reporting for customers than the one your in-house testing team uses - the latter might include a “testing phase” field and/or an “issue open/closed” toggle.  Regardless of the specific defect management process flow, no defect report should ever marinate too long in your system or, even worse, disappear into the digital netherworlds!

The primary purposes of defect reporting and tracking include:

  • Record and track defects with the system under test

  • Provide developers, testers, and anyone else addressing the defect with enough information to reproduce and find root cause(s), and eventually fix the problem

  • Provide developers and test managers a way of tallying metrics on defects and ultimately improving the test process and quality of testing

   

The end goal of course is to continually improve the system under test:  The user journey through the system should be smooth and productive, and the experience should only improve over time due to timely fixes and feature enhancements.

The more informative, detailed, and clear the defect report then the less time the testers, developers, or whoever is charged with acting on it have to spend deciphering and trying to understand the defect in question (see good and bad examples below).  Very often a bug that can’t be reproduced will not be fixed.  Also, the focus should be on reporting the defect and related details - and not in any way a dissertation on how the developers should have built the system under test (aka "smug report").

This might be a controversial opinion...but the more concise the defect report template, then the more likely it will actually be filled out in a timely manner.  Everyone is busy and often doing multiple jobs these days, and so it’s easy for someone to look at a dauntingly gargantuan defect template and procrastinate or even let it slip through the cracks.  We want fast feedback on our system defects, and so any stakeholder who comes across a problem should submit their defect report as soon as possible.  For those folks who love to be meticulous and/or overshare – and both are a good thing when it comes to defect reports – they can simply add more information in the “description” field.

Defect Report Template:

Defect ID: Unique identifier associated with each defect; your defect management system should automatically generate this

Date: Your defect management system can populate the date, but it should allow the reporter to manually change this if the defect occurred a day or two earlier and they are reporting it late

Severity: There should be some form of scale from minor issue to a reverse big bang where the Universe collapses on itself

Reporter Name: This should be populated automatically, linked to defect management system login

Platform/Environment: Testers can get so wrapped up in the weeds of how the defect occurred, that they can forget key environment information; hardware information beyond brand and/or CPU maker can get overly in-depth and so I would leave it up to the reporter as to how much of that to include

Any Additional Dependencies: Worth noting if additional variables are involved, like if applications or SDKs related to the issue were installed

Build Version: The version number is absolutely critical for reproducing and fixing the defect

Description: Summarize everything related to the defect, and add any relevant information not covered in other parts of the defect template

Expected Result: This should be kept strictly to what is expected by the system within the boundaries of the user requirements, and NOT a lengthy manifesto on what the reporter would prefer the developers have done instead (that's a Smug Report)

Actual Result: The reporter should be very specific to note not just the defect, but anything else unexpected or weird

Steps to Reproduce: Think of this as the “defect reproduction recipe” for anyone reading the report

Artifact Attachment(s): This includes screenshots, log files, and anything that could be used for reproducing, debugging, and testing; consider making this field prominent so a defect reporter in a hurry doesn’t miss it

Good Example:

Defect ID: DFT82321339

Date: 8/6/20

Severity: Level III

Reporter Name: Otto Mation

Platform/Environment: Chrome v.84.0.4147 in Windows 10 v.2004 on a newer Dell w/ Intel CPU

Any Additional Dependencies: None

Build Version: PomeloShop Version 3.2

Description: This issue was found when doing exploratory testing on the recently revamped customer portal; all portal login test cases will eventually be automated tests.  Logging in with valid credentials to the Pomelo Shop login portal at “http://www.pomeloshop.com/index.php?route=account/login” results in an error message rather than the welcome screen.  Clicking OK on the message refreshes the login page.  I recall we experienced a similar error in version 2.2 when running the Safari Browser; worth checking previous defect records for similarities when debugging.

Expected Result: The user should be taken to the Pomelo Shop exotic fruit store returning customer welcome screen

Actual Result: Error message pops up stating “Your login information was found, but is somehow corrupted”

Steps to Reproduce:

  1. Enter valid username “otto.mation@tryonsolutions.com
  2. Enter valid password “OttoPass123”
  3. Click Login button

Artifact Attachment(s): Screenshot

Bad Example:

Defect ID: DFT82321339

Date: 8/4/20 (late reporting)

Severity: Kinda Bad

Reporter Name: Otto Mation

Platform/Environment: My favorite browser in Windows

Any Additional Dependencies: Installed some stuff from sketchy gambling sites

Build Version: The earlier PomeloShop one

Description: Login no work

Expected Result: The current Pomelo Shop fruit store welcome screen is boring; I want a fancy one with dancing exotic fruit mascots that shimmy and twist around the user’s screen

Actual Result: Some error popped up over top of the current boring welcome screen

Steps to Reproduce:

  1. Put stuff into login boxes
  2. Click button

Artifact Attachment(s): Nope

 

I don’t profess that the above template is perfect for all systems to be tested, types of users doing the reporting, or defect management processes, but I think at the very least it could be used as a solid starting point to expand upon.  How does this template compare to what you use to report defects?  Drop us a line and let us know!

This post was written by:

James Prior
Technical Pre-Sales Consultant

James has been working in software pre-sales and implementation since 2000, and has more recently settled into focusing on technical pre-sales. He takes care of our hands-on demonstrations, and eagerly awaits your request to see our Cycle test automation software in action. Drop him a line at: james.prior[at]tryonsolutions[dot]com.

Recommended Content

 

 

Global Engineering Group Upgrades WMS with Tryon Solutions & Cycle

Global Engineering Group Upgrades WMS with Tryon Solutions & CycleOur customer is a global, high-technology engineering group founded in 1862. The company is a leader in tools and tooling systems for metal cutting; equipment, tools and services for the mining and...

Checklist: Deciding which Tests to Automate

Essential Checklist for Deciding Which Tests to Automate To automate or not to automate?That is the question a lot of test professionals ask themselves- when deciding what tests should be run by software and which ones are best performed by people?To help, we've come...

Cycle & Behavior Driven Development: How It Works Demo

What We Covered: High level explanation of Cycle as a behavior-driven test automation solution Overview of advantages including collaboration, lower maintenance, streamlining, and how Cycle is used to mitigate deployment risk Overview of Cycle interface, syntax, key...