Creating an Analytics Culture of Trust
In a recent report, consulting firm KPMG tackles a pair of thorny questions. Why don't decision makers place greater trust in data and analytics and how do you build this trust?
In its October 2016 report "Building Trust in Analytics: Breaking the Cycle of Mistrust in D&A," consulting firm KPMG tackles a pair of thorny -- and particularly timely -- questions.
First, why don't executives place greater trust in data and analytics? Second, and just as important, how do you build this trust -- especially in the context of a legacy of data or analytics failure?
According to KPMG, a surprising number of organizations lack confidence in their data and analytics insights. "Given the power that it holds, trust in D&A [data and analytics] should be a nonnegotiable business priority. Yet our survey reveals that this may not be the case. In fact, 60 percent of organizations say they are not very confident in their D&A insights."
These results aren't all that surprising given that a bad experience with business intelligence (BI) or analytics can bias an entire decision-making culture against its use. This isn't to say that business decision makers completely discount BI and analytics insights, but bad experiences can poison the proverbial well. At the very least, they can cause businesspeople to resist using BI and analytics to aid decision making.
Whether the cause is BI tools that are hard to use, failed data warehouse projects, or reports, dashboards, and visualizations that give divergent or inconsistent results -- many executives, directors, managers, and other decision makers have been burned by BI and analytics.
Fool Me Once...
That 60 percent is just the top-line number. The internals of the KPMG report should sound familiar to any data management professional.
Just 10 percent of respondents believe they do a great job managing the quality (i.e., accuracy, consistency, and timeliness) of their data and analytics. Slightly more (13 percent) are confident in their ability to safeguard data and analytics privacy -- or to make ethical use of data or insights. Worst of all, just 16 percent think they do a good job vetting the accuracy of their analytics models.
In a way, organizations are setting themselves up for failure with analytics. For example, just 46 percent agreed with the statement that their "analytics and model-building techniques aspire to meet industry best practices and standards." Slightly fewer (45 percent) said that they "consistently use rigorous quality checks to ensure the accuracy of data and analytics models and outputs."
When it comes to vetting analytics models for accuracy, U.S. companies fare better than those in other countries. More than half (53 percent) said they validate their models with third parties, while 60 percent said they're confident in the capabilities of their data analysts. KPMG didn't provide a region-by-region breakdown for this data point. It did note that in France and Germany, far less than half (37 percent and 38 percent, respectively) of organizations say they enforce rigorous quality checks on data and analytics. Similarly, less than 40 percent of organizations in Canada, China, France, Germany, and South Africa say they're confident their analysts have selected the appropriate source data for analysis.
On a global basis, it gets worse from there. KPMG finds that "less than 40 percent of analytics teams work with business partners to set objectives up front. This means that many analytics teams may be working in their own silos without truly linking their activities back to business outcomes."
Critical issues that used to bedevil DBAs and developers in the traditional data warehouse model haven't gone away. The report claims "just 45 percent of organizations say they rigorously check the quality of their data and even fewer believe that they always select the right internal data sources."
Quality Is the Question
The lack of quality control makes sense: with more businesspeople using self-service tools to bypass the data warehouse and BI tools stack, it becomes that much more difficult to control for data quality and data lineage/sourcing.
Between Tableau-toting analytics discoverers and users who spin up their own (sanctioned or unsanctioned) DBMS sandboxes, the distinctive features of the data warehouse -- that it comprises a single system of record and offers a single, centralized means of access -- have ceased to matter, at least to many users.
Hard-to-use BI tools and a slow, inflexible, overcentralized data warehouse architecture have eroded trust in analytics. The shift toward self-service BI and analytics, however, won't redeem this trust; if anything, it's poised to make matters worse.
This problem is compounded by the challenges of advanced analytics, as KPMG observes in its report. "The quality of analytics poses huge potential trust issues. Statistical and algorithm design, model development approaches, and quality assurance are becoming critical ... organizations are struggling to assess quality in scenarios in which the impact of low quality can be high or where there is no known right answer with which to compare the output of a new decision engine.
"As analytics move into critical areas of society, such as decision engines for drug prescribing, machine learning 'bots' as personal assistants, and navigation for autonomous vehicles, it seems clear that [data and analytics] quality is now a trust anchor for everyone."
A Prescription for Action
How do you promote trust in data and analytics? KPMG has a few suggestions.
First, establish cross-functional -- multidisciplinary -- data and analytics teams. This means involving key stakeholders from data management and analytics along with business and IT people.
Create a meta-model -- a model of models, in KPMG's parlance -- that makes it possible to understand how changes in one widely used variable will affect all extant models. "This meta-model can also help ensure consistency in how data is used across different analytical models and can help executives prioritize projects that will deliver the highest value to the business."
Make analytics more transparent. Black boxes are no good and KPMG cautions that "data scientist competitions ... mandate the release of the winners' analytics for peer review and refinement. Regulators force this approach in some markets ... to make sure that every single element is understood, reviewed, and re-reviewed because of a potentially critical impact on the world economy." These regulations are in place for a reason, after all.