Sweat in Testing, So You Don’t Bleed in Go Live

The importance of testing can’t be understated, especially by anyone who’s been on the front-line of a botched software implementation.  The ripple effects of a major defect experienced during a customer Go Live are varied and many.  In the early 2000s, I was part of a team that installed an ERP and quality control solution on-site at a textile mill. This company was a priority customer, and so in-turn they were first-in-line to receive our big, new, and shiney software. The product worked well for about a month and then…the database bombed because it couldn’t handle the load. The developers had chosen Microsoft Access as the backbone, rather than a more capable and robust option like SQL or Oracle.  Long story short, the customer was furious and had to be rolled back to an ancient (but stable) version.

There are plenty of numbers I could cite like man-hours wasted and extra costs incurred or something more granular like defect removal efficiency and cycle time. Instead I want to go beyond the numbers and highlight the internal and external “reputation fallout” that could have been prevented with a simple load test.

Management no longer trusted the developers. Our development team had previously enjoyed autonomy and, after the debacle, they were given rigid deadlines and had to regularly report detailed status updates to the department Vice President. The increased oversight was permanent.

The customer no longer trusted us. Prior to the screw-up, anyone from my company could waltz in to their office, exchange hi-fives, and then install an upgrade or apply a patch.  This changed drastically. Support and implementation staff now had to convince the customer’s head of IT that nothing they did would result in a crash.  We were also told that they didn’t want any more major updates; only approved incremental patches to the old “trusty” version of the software. I was one of a handful of people that still frequently visited this customer, and let’s just say that I’m glad we didn’t track metrics for snide comments by personnel burned from botched implementations.

News traveled fast in the close-knit industry.  We heard through the industry grapevine that other existing customers were worried about “being used as a beta site”, and undoubtedly more than a few potentials had caught wind of the fiasco as well. One bad deployment had tarnished our previously stellar reputation.

It was nearly a decade before our good name was restored. One quick and easy database load test. The loss of money, data, AND reputation could have been prevented If we had done just ONE SIMPLE LOAD TEST pushing a million records through the system.

I’ve made the case for testing in general, but of course it’s also important that you do the right testing. Maximize your time performing exploratory testing and designing test cases, and minimize repetitive manual testing, test case maintenance, communication errors, and developer/coder involvement. To do all of this effectively, you and your team should have a behavior-driven test automation solution in place. Feel free to contact us to discuss our test automation solution called Cycle, or even just to pick our brains on software testing in general.

This post was written by:

James Prior
Technical Pre-Sales Consultant

James has been working in software pre-sales and implementation since 2000, and has more recently settled into focusing on technical pre-sales. He takes care of our hands-on demonstrations, and eagerly awaits your request to see our Cycle test automation software in action. Drop him a line at: james.prior[at]tryonsolutions[dot]com.