Software Bugs Cost Billions, Study Says

A U.S. Department of Commerce study has concluded that software bugs, or errors, cost the U.S. economy an estimated $59.5 billion annually, but not everyone agrees that the blame lies entirely with software vendors.

According to the study–commissioned by the Gaithersburg, Md.-based National Institute of Standards and Technology (NIST), a part of DOCat the national level, more than half the costs of such errors are borne by software users, with the remainder falling on software developers and vendors.

Released in June, the study was funded by NIST and conducted by the Research Triangle Institute in North Carolina.

The study”The Economic Impacts of Inadequate Infrastructure for Software Testing”also found that although not all errors are likely to be removed, more than a third of these costs ($22.5 billion) could be eliminated by an improved testing infrastructure that enables earlier and more effective identification and removal of software defects by vendors.

“These are the savings associated with finding an increased percentage (but not 100 percent) of errors closer to the development stages in which they are introduced,” said NIST. “Currently, over half of all errors are not found until downstream in the development process or during post-sale software use.”

“The impact of software errors is enormous, because virtually every business in the United Stated now depends on software for the development, production, distribution and after-sales support of products and services,” stated NIST Director Arden Bement.

According to NIST, software is “error-ridden” in part because it is growing in complexity.

“The size of software products is no longer measured in thousands of lines of code, but in millions,” the agency said. “Software developers already spend approximately 80 percent of development costs on identifying and correcting defects, and yet few products of any type other than software are shipped with such high levels of errors.”

Indeed, if software companies were automobile makers, “they might be out of business from product liability suits,” according to Gregory Tassey, senior economist at NIST. Software “has way more errors in what is delivered to users than the vast majority of products you run across.”

For the study, two industriestransportation equipment and financial serviceswere examined in depth, said NIST. In financial services, data were collected from four industry software developers, as well as 98 users (primarily banks and credit unions).

According to NIST, about two-thirds of the users reported experiencing “major software errors” in the previous year.

Major errors, said Mr. Tassey, include those that result in systems shutdown, loss of data, or need for significant systems reconfiguration.

Respondents who did have major errors reported an average of 40 major and 49 minor software bugs per year in their clearinghouse software systems, said NIST. Typical problems encountered due to bugs were:

Increased person-hours needed to correct posting errors.

Temporary shutdown leading to lost transactions.

Delay of transaction processing.

NIST estimated the total cost of inadequate software testing in financial services to be $3.3 billion. Potential cost reduction from “feasible” infrastructure improvements is $1.5 billion.

Mr. Tassey asserted that software-error-related loss scenarios are “probably true of software across all industries.” He said the highly publicized software failuressuch as one that interrupted the New York Mercantile Exchange and phone service to several East Coast cities in February 1998″are the tip of the iceberg.”

Not all of the fault for software errors lies with vendors, however, according to Eli Dabich, president of Synergy 2000, a Pasadena, Calif.-based systems integrator serving the insurance industry. He maintains that companies that buy software “bear equal responsibility.”

In a typical scenario, said Mr. Dabich, a company buys software from a vendor who promises to put out one new release, or upgrade, a year. “But it probably takes two years to install the system, so most companies will elect to wait for release three,” he explained. The company may also decide that release four isnt that much different than three, so they never install it.

The result is that companies may miss fixes and other necessary add-ons that could eliminate or mitigate the effects of errors, said Mr. Dabich.

In addition to not staying current with updated releases, buyers will also “tamper” with the basic code of the program in order to adapt the software to its existing workflow processes, Mr. Dabich noted.

Revisiting the automobile analogy, Mr. Dabich said that when one buys a car, one knows it has been tested by the manufacturer to work as it has been built. “Youre not going to screw with the basic car the way you would with a software package,” he stated. “But youll tailor the package to the way you want to work.”

Instead, Mr. Dabich recommends that software buyers tailor their processes to the software they purchase. This would “cut down bugs, because youre not playing with the system,” he asserted.

When it comes to who is responsible for software errors, Mr. Dabich said, “Id be willing to bet that its 50-50 between the vendors and the buyers.” While buyers arent keeping up with current releases or are changing basic programming, “vendors are not fixing problems fast enough and theyre not making it easy to implement new releases,” he stated.

Why do buyers willingly accept products that contain so many errors? According to Mr. Dabich, senior company officials are often unaware of the magnitude of the problem.

“Most CEOs dont know how messed up their IT is,” stated Mr. Dabich. Part of the reason for that is that information technology professionals are reluctant to admit that they are spending time fixing bugs in software that IT may have recommended purchasing, he explained.

According to the NIST study, “all developers of financial services software agreed that an improved system for testing was needed. They said that an improved system would be able to track a bug back to the point where it was introduced and then determine how that bug influenced the rest of the production process.”

The study added that the developers believe that better testing tools and methods could reduce installation expenditures by 30 percent.

NIST stated that the development of standardized testing tools “that have undergone a rigorous certification process would have a large impact on the inadequacies currently plaguing software markets. For example, the availability of standardized test data, metrics and automated test suites for performance testing would make benchmarking tests less costly to perform.”

Reproduced from National Underwriter Property & Casualty/Risk & Benefits Management Edition, July 29, 2002. Copyright 2002 by The National Underwriter Company in the serial publication. All rights reserved.Copyright in this article as an independent work may be held by the author.

Special Reports /

Software Bugs Cost Billions, Study Says

Related Stories

Recommended For You