Mission Critical Software Needs Better Developer Testing

By Avi Hein (Profile)
Thursday, October 25th 2012

What is the cost of a software bug? For Knight Capital, an American global financial services firm, it was at least $440 million due to a glitch in their legacy software. The loss was caused by a failure in an update meant to accommodate a new New York Stock Exchange (NYSE) system that went live on August 1. $440 million was lost in only 30 minutes, before it was manually shut down. That is not counting the additional damage caused to their reputation and stock valuation. This is not insignificant – Knight Capital is not some fly-by-night startup in a garage but rather the largest trader in American equities.

Catching the issues early, during development and preparation for integration with the new NYSE system, which can only be done by adequate and comprehensive developer testing, and not a separate QA team that doesn’t examine the code until the developer is done, would have prevented this loss.

Knight’s glitch wasn’t the only technology failure we recently saw in the financial sector. NASDAQ OMX was forced to pay $62 million in a settlement compensating investors for the delay in trading Facebook shares during its IPO, caused by a “technical glitch.” BATS Global Markets was forced to cancel their IPO due to a “software bug” in their trading platform.

This time, NASA got it right with August’s launch of the NASA Mars rover that is now providing valuable scientific information about Mars. Last time NASA tried to launch a vehicle to Mars, however, things did not go as planned. In 1999, NASA lost a $125 million orbiter because one engineering team used metric units while another used English units for a key spacecraft operation. In the follow-up report, Edward Weiler, NASAs Associate Administrator for Space Science said, “The problem here was not the error, it was the failure of NASAs systems engineering, and the checks and balances in our processes to detect the error. That’s why we lost the spacecraft.” The lack of adequate testing in mission critical systems has a real cost.

Unfortunately, due to the perceived difficulty of testing or added cost up-front (more than reaped back by the faster time to release and improved quality after release), too many organizations view software testing as the responsibility of a separate quality assurance team, instead of first and foremost, the job of the developers. Developer testing puts the burden of software quality on the software author – the developer. It ensures that the developer’s code does what he or she intended it to do and lets companies catch bugs early – before deadlines get too tight and damage is done.

The traditional way of relegating testing to a separate team is over. According to Forrester, “development testing has dramatically increased in importance.” The longer defects remain in code, the higher the cost. Developer testing catches the defects quicker than waiting for a separate testing QA team to find the defects and play ping-pong with the development team.