Poor Data Quality: A $600 Billion Issue?

Despite these dangers, however, "most executives are oblivious to the data-quality lacerations that are slowly bleeding their companies to death," stated the report, entitled: "Data Quality and the Bottom Line." The report also found that nearly 50 percent of survey respondents had no plans to implement measures to improve data quality.

The reports findings were based on survey results from 647 respondentsprimarily U.S.-based information technology managers and staffacross a broad range of industries. Approximately 11 percent of respondents were in financial services companies, while another 9.5 percent were in insurance firms.

According to Wayne Eckerson, director of education and research for the Data Warehousing Institute, the definition of data quality is simply removing errors found in data. Such errors are often the result of data entry mistakes, such as transposing letters or numbers. In addition, data are often wrong due to changes over time, such as when an individual dies, he noted.

"Its also how you interpret the data," Mr. Eckerson pointed out. Within the same organization, different divisions might have different definitions for the same data elements or terms. Classic examples, he said, are what constitutes a "sale," or how "gross profits" are calculated.

"The problem with data is that its quality quickly degenerates over time," the report stated. "Experts say 2 percent of records in a customer file become obsolete in one month because customers die, divorce, marry or move. In addition, data-entry errors, systems migrations and changes to source systems generate bucket loads of errors."

"Truthfully, though, most data quality problems come up with the name and address data fields," said Mr. Eckerson, who characterized the cost of mislabeled, misprinted communications as "astounding."

The report cites a "real-life" insurance example in which a carrier receives two million claims per month with 377 data elements per claim. "Even at an error rate of .001, the claims data contains more than 754,000 errors per month and more than 9.04 million errors per year," the report noted. "If the insurance company determines that 10 percent of the data elements are critical to its businessthe firm must still fix almost one million errors each year."

Further, the report noted, if the insurer estimates $10 per error in staff time needed to make corrections for erroneous payouts, etc. (a "conservative" estimate), "the companys risk exposure to poor-quality claims data is $10 million a year."

The report added that the estimated risk in the example does not include the firms exposure to poor data quality in financial, sales, human resources, decision support and other operations.

What can a business do to reduce its risk?

While there are some technology tools that will help, Mr. Eckerson said the key is to be committed to keeping data clean on a continuous basis. "A lot of companies put a Band-Aid on [data-quality problems] and they get by," but it becomes a problem when it affects a high-profile project or operation, he noted.

"Data is as critical a resource as a companys assets and money," stated Mr. Eckerson. For that reason, he added, "companies need to standardize processes for managing data."

The report recommends that firms "treat data as a strategic corporate resource, develop a program for managing data quality with a commitment from the top, and hire, train or outsource experienced data-quality professionals to oversee and carry out the program."

Commercial data quality tools and service bureaus that automate the process of auditing, cleaning and monitoring data might be worth the investment, the report added.

"Most commercial tools are now moving beyond auditing and scrubbing name and address data to tackle other data types," according to the report. "They are also beginning to step up to the challenge of validating company-specific business rules."

The bottom line, said Mr. Eckerson, is to embrace "total quality management for data," which means stopping data errors at their source.

The Data Warehousing Institute is an independent association involved in education, research and training for data warehousing and business intelligence professionals. The data-quality study was sponsored by six "leading data-quality solution providers"–Arkidata Corp., DataFlux Corp., DataMentors Inc., Sagent Technology Inc., SAS Institute and Vality Technology Inc.

For more information about the study, go to www.dw-institute.com.

Data Quality Attributes:

The Data Warehousing Institute in Seattle lists the following questions to pose when assessing data quality:

Accuracy: Does the data accurately represent reality or a verifiable source?

Integrity: Is the structure of the data and relationships among entities and attributes maintained consistently?

Consistency: Are data elements consistently defined and understood?

Completeness: Is all necessary data present?

Validity: Do data values fall within acceptable ranges defined by the business?

Timeliness: Is data available when needed?

Accessibility: Is the data easily accessible, understandable and usable?

Reproduced from National Underwriter Property & Casualty/Risk & Benefits Management Edition, March 18, 2002. Copyright 2002 by The National Underwriter Company in the serial publication. All rights reserved.Copyright in this article as an independent work may be held by the author.

Contact Webmaster