TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

Avoiding Data Discord in Your Enterprise

The vortex of anger, criticism, and abuse that has engulfed the Web must not be allowed to pollute business decision making.

• By Barry Devlin
• December 20, 2017

This year we captured more data than any year in history. Gartner projects that the number of IoT devices has exceeded that of people on Earth this year. In social media, Facebook exceeded two billion monthly active users in the third quarter, only five years after it broke the billion mark.

For Further Reading: The New Ethics of Data Management The Shortcomings of Predictive Analytics Algorithms or Democracy -- Your Choice

By any reasonable reckoning, we have enough data to improve not just business but society beyond our wildest dreams. Promoters of BI, analytics, and big data have for decades claimed that more information leads -- inevitably -- to better decisions. In business, evidence remains patchy if you count the number of failed BI projects. In society at large, life-threatening environmental and communal decision making in 2017 seemed to occur in a fact-free vacuum. Where have we gone wrong?

On the Origin of Facts

For more than two centuries, progress in most of society and business has been built on a foundation of facts, scientific methods, engineering, and -- most important -- shared context and rational discourse around facts.

Facts about the real world are discovered, proven, or disproven via scientific research. Theories are constructed around these facts, enabling engineers to design and build the devices, tools, and structures on which modern society and business depend. Few, if any, reasonable people would disagree. Nonetheless, flat earth advocates and climate change deniers believe and promote views that fly in the face of almost the entire scientific community. How can this be?

The reason lies in the destruction of shared context and the decline of rational discourse driven, in large part, by the use, misuse, and abuse of the Web over recent years and most obviously in the last two years. Facts -- or, more precisely, useful facts -- mainly consist of derived information and conclusions built upon measured data and agreed to via shared context and rational discourse. It is only through the derived information and conclusions that decisions can be reached and action taken.

For example, measured data shows rapid growth in atmospheric CO2 concentration and global temperature. Scientific experiment in the laboratory has long proven cause and effect. Very few people claim the data isn't accurate. Most of the dispute is more subtle, attacking the derived information and conclusions -- e.g., the changes are not manmade, the trends are reversing, nature can adapt -- a recent scientific study showed how this misinformation can spread. Based on this level of understanding, action is taken -- or not.

The reality is that which sources and people we believe and trust for such information and conclusions are strongly influenced by our social groups. This choice has been heavily influenced for more than half a decade by the filter bubbles blown by social media giants such as Facebook and Twitter, mostly to feed their appetite for advertising revenue.

Furthermore, believing that a particular piece of information is true depends on accepting the expertise of the people who promote it. Identifying "real" experts on the Web where all pages are notionally equivalent but popularized by emotional biases, misinformation, and propaganda has become well-nigh impossible.

The Data Dilemma for the Enterprise

Business decision making is not immune from these societal issues.

The use of social media data in sentiment analysis can be skewed by the filtering and targeting behavior of major Internet search, social media, and service providers. IoT data is just data; information about driving behavior feeding pay-as-you-drive insurance applications emerges only from the context and interpretation of basic data about events and measures in the vehicle and combining it with data from other sources.

Decision makers are subject to as wide a range of biases in data interpretation as is the general public. Business facts emerge from social media and IoT data through statistical analysis, modeling, and algorithms that may range from balanced (though unintentionally biased) to deliberately manipulated. "Fake news" exists in business as well as in politics.

These issues cannot be overcome with technology, although it may help us identify the size and scope of the problems. Decision makers must take a deep look at the social and mental processes underlying their analyses and actions.

The vortex of anger, criticism, and abuse that has engulfed the Web in the past year must not be allowed to pollute business decision making. In the face of growing societal discord, senior executives must take the lead by carefully cultivating social inclusiveness, context building, and reasoned discourse within their teams and across business boundaries.