LESSON - Controlling Data Quality to Improve Business Decisions
By Tony Fisher, President/General Manager, DataFlux
Anyone who has ever flown a plane—or even glanced into a cockpit when boarding a commercial flight—can appreciate the complex array of gauges and monitors that the pilot must check. All the data about a plane’s speed, course, fuel, and other details are in easy view, each giving the pilot the information necessary to make sound, safe decisions.
Similarly, organizations rely on data to provide the foundation for business decisions. For years, companies have implemented business intelligence (BI) programs to achieve one goal: to make better decisions from their corporate information. Many companies have discovered one inescapable truth: it’s impossible to make an informed decision based on outdated or erroneous information. Just as a pilot needs to monitor the health of the aircraft, organizations need to constantly gauge the health of their data.
Building and keeping good data on customers, prospects, products, and inventory takes constant vigilance. To manage data effectively, an organization must institute a data management program based on continual, routine monitoring of data to increase the control on data quality.
Companies often believe that after cleaning data once, they have solved their data problems. However, building and keeping good data on customers, prospects, products, and inventory takes constant vigilance. To manage data effectively, an organization must institute a data management program based on continual, routine monitoring of data to increase the control on data quality.
“Once and Done” is Not Enough
The impact of “data decay” can influence—and hinder—many enterprise initiatives. Imagine a manufacturing company that builds a data warehouse to serve as a single repository for all of its information about customers, products, and inventory. From that data, they can uncover trends about customer adoption, resource allocation, and future needs.
After a review of the data, this company finds that new, non-standard information is constantly arriving at the repository. The effect of this bad data may not be felt until much later. Whenever the company explores this data to identify patterns or tendencies, the presence of bad data can skew the results.
The solution for building high-quality corporate data on an ongoing basis is data monitoring. With data monitoring, technology and business users can create rules to examine data automatically to uncover problems as they occur. These users can also chart metrics related to data quality on a periodic basis and begin to address some of the underlying reasons that bad data is being collected in the first place.
The Role of Data Monitoring
The role of data monitoring is similar to quality improvement methodologies like Six Sigma. Instead of loading questionable information into a data warehouse, data monitoring puts checks and controls on incoming information to keep high levels of data quality.
A data monitoring regimen can accomplish a number of tasks, such as:
- Detect problems from incoming data. Since data warehouses typically receive periodic loads, this allows companies to validate existing data against established business rules. This will help them to uncover and address data integrity issues—before they become a problem later during business intelligence programs.
- Generate instant alerts. Set up automated system notifications and e-mails to flag problematic data as new, inconsistent records enter the system.
- Identify trends in data quality metrics. View ongoing statistics about data to see when the value of data starts to decline.
Data monitoring extends the reach of traditional data quality programs by making good data a corporate priority. When data does get out of control, users know immediately—and they can react to problems before the quality of the data declines.
Add Monitoring to Data Management Initiatives
Building consistent, accurate, and reliable data is not easy. Periodic fixes will only provide temporary relief from the various problems that can arise because of bad data. With data monitoring, companies can better control their data and build more reliable information to support any future business intelligence efforts.