TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

Correlation, Causation, or Simply Gut Feel?

Surveys are often used to show correlations, but don't assume this means there's a cause-and-effect relationship.

• By Barry Devlin
• October 7, 2019

The title of a July Deloitte Insights article Analytics and AI-driven enterprises thrive in the Age of With -- The culture catalyst piqued my interest. Despite the oddly named "Age of With," I thought the report might offer firm evidence of the business value of analytics and AI in digital business. The following passage, among others, left me disappointed:

For Further Reading: Beer and Diapers: The Impossible Correlation The Secret to Organization Data Science Success: Data Literacy Analytics is Only One Scenario in Decision-Making Support

The survey [reveals] a strong correlation between culture and business performance: Organizations that reported having the strongest cultural orientation to data-driven insights and decision-making are twice as likely to have reported exceeding business goals in the past 12 months. Forty-eight percent of these businesses say they outperformed their target versus just 22 percent of those with a more diluted analytics culture.

The report was authored by Deloitte staff members Tim Smith, Ben Stiller, and Jim Guszcza in collaboration with well-known analytics and digital business professor and author Tom Davenport.

The quote bears re-reading a couple of times. The accompanying graphic entitled, "The link between insight-driven culture and business performance" is also worth a look. The authors correctly and carefully note a correlation (and perhaps less carefully in the graphic, a link) between insight-driven culture and business performance.

Casual business readers, lacking in statistical knowledge of the difference between correlation and causation, might be forgiven for assuming that implementing an insight-driven culture in their organization might lead to better business results. Of course, they would be mistaken in this belief, and the authors could easily defend themselves against such an interpretation. Nonetheless, the tone and, indeed, the title of the article imply that cause and effect may occur here.

Where Lies the Truth?

The challenge with surveys is that the questions asked are designed to capture a relatively limited set of variables that try to explore some hypotheses of interest to the researchers. In this case, the authors are clearly interested in why only a minority of companies have become insight-driven despite a decade of emphasis on big data and analytics. Their hypothesis is that organizational culture may be the impediment to this (assumed to be) worthwhile goal.

Anecdotal evidence and my own personal experience suggests that culture is at least a partial contributor to the limited uptake of analytics in organizations. However, I am also aware of situations where funding issues, data availability and quality challenges, and the lack of suitably skilled staff have slowed the use of analytics. These hypotheses, like all hypotheses, are based on gut feel. Despite data-driven myths to the contrary, all theories about how the world works begin with the intuitions or gut feel of knowledgeable people in the field or, indeed, outsiders who bring novel perspectives.

With only a limited view of either the questions or the base data provided in the article, it is impossible for the reader to know if any of contrary or null hypotheses were evaluated by the study. This leaves statements such as the above quote subject to serious doubt. The challenge the Deloitte article poses is that survey-based conclusions must be written carefully and read critically to avoid unjustified assumptions about causality.

In fact, organizational culture is not easy to measure. The study apparently relied on self-identified positioning on an "Insight-Driven Organization Maturity Scale" as a basis for evaluating culture (and leadership) for analytics in the organization. In contrast to the evidence adduced in the opening quote, the direct correlation between position on this scale and business success is rather poor, as closer reading of a later figure in the report shows.

Other interesting correlations are called out by the article. For example, the survey finds that only 18 percent of respondents take advantage of unstructured data, versus 64 percent who rely on internally sourced structured data. This observation is interesting in itself (why is the use of big data still so low?) but is used to set up another point correlation that "[e]xecutives who say unstructured data is one of the most valuable sources of insights are 24 percent more likely to have exceeded their business goals." Possible causes of this eye-catching correlation are, unfortunately, not offered.

The bottom line for me is that the authors began with a gut feel that organizational culture was an important factor in successful analytics implementations. Their survey provided some interesting correlations between self-reported cultural maturity and business results. However, causality was neither proven nor reported, and readers should not assume that any cause and effect was at work here.

Surveys Offer Limited Insight

Although I've focused exclusively on the article by Davenport and his colleagues at Deloitte, the issue is much broader. With the popularity of the concept of "data driven," surveys are often seen as providing numerical proof for particular ideas. This is seldom the case.

I, too, participated in running surveys of big data uptake from 2012-16 and tried to extrapolate from survey data to find support for my gut feel about the reasons for specific behaviors and outcomes in the industry. I often struggled to rationalize basic data that I considered counterintuitive and to discover correlations of interest.

As BI purchasing decisions move from IT to business, there is a growing urge among IT managers and software vendors to find measurable business justification for investment. This requires an understanding of causation. As a consequence, survey data and correlations it reveals should be treated with a grain of salt -- and not as cause-and-effect conclusions.