TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

The Future of Analytics: Illuminating the Modern Workplace

Now that companies must keep a larger amount of unstructured data, they're looking for ways to leverage it, opening the door for a paradigm shift in analytics. Here are three predictions about the development of analytics over the next few years.

• By Kon Leong
• October 14, 2016

I'm often asked in which area of business I expect analytics to make the biggest strides in the near future, and my answer is pretty simple, although perhaps counterintuitive: it will be within the workplace itself. Endless resources are spent on gathering consumer data -- and rightfully so -- yet to many companies, the dynamics of their own workplace remain a mystery.

It isn't for lack of data. Written communication is difficult to analyze because it is unstructured, making it inherently resistant to quantification. Without a clear incentive to save this data, companies in the past have treated it as a burden rather than a resource and left it largely unused.

However, within the last decade or so, increased regulatory pressure has propelled companies to begin storing what was once considered "dark data." Now that they must keep it, they're looking for ways to leverage it as a resource, and thus the dark data has opened the door for a paradigm shift in analytics. This is what you can expect in the development of analytics over the next few years:

#1: Reactive Analytics Will Fail

Analytics is hot right now, but most of the new players use what I call "The Sandbox Approach." They wait until something compels them to take a closer look at their data, perhaps a security breach. Then they begin collecting data from the people involved to put it in a virtual sandbox where they determine what was taken and where the attack came from. However, when the entire beach of data is unmanaged, so is the data that goes into the sandbox.

It is only once all data across the enterprise is proactively managed, and human-driven analytics and governance begin working in synergy, that analytics will live up to its full potential.

One use case for this relationship is in file analysis: every piece of data within an enterprise can be tagged according to sensitivity, providing companies with a blood flow diagram of where important information lies. Companies can then identify irregular data usage -- abnormal amounts of sensitive data being transferred or data being transferred to unexpected locations -- and neutralize threats before they actually materialize.

Once companies begin proactively managing their employee-made data, they will be able to use analytics for behavioral applications that transcend data security. Which types of organizational networks are most effective? How is information distributed throughout the company? How can interdepartmental processes and communications be improved? What patterns of behavior can we identify in the most effective employees and train other employees to imitate?

These are questions that unstructured data such as file shares, email content, and metadata will help companies answer in the coming years.

#2: Analytics and Information Governance Will Clash

Although analytics and governance must work in synergy, the two actually share a complicated -- in many ways antithetical -- relationship. Governance focuses on data with an eye toward risk exposure, and that has led to a minimalist philosophy towards data storage: the more data we can delete, the smaller the security risk.

Analytics, on the other hand, incentivizes storing more information and for longer, a strategy that undermines the protections of governance. Despite the many ways they complement one another, their philosophies are ultimately at odds.

The question is: Would you rather your corporation have impenetrable governance (even if it means handicapping the performance of analytics) or would you rather take a chance at driving revenue at the risk of security?

Answering this question will likely spark countless arguments, but ultimately the visionaries will win. Information governance will be charged with mitigating the risk presented by aggressive analytics strategies, and corporations will have to adapt accordingly and challenge precepts in governance.

#3 Control of Information Will Threaten Our Privacy, but Governance Will Restore It

I find it strange that we're apathetic towards information control despite the potential for abuse. If you look at the development of nuclear power -- the only comparatively powerful force that comes to mind -- its potential was immediately recognized and regulations were put in place to check it. Yet when it comes to the treatment of our personal information, we remain indifferent.

With today's technology, a dictatorship would be able to trace every piece of information about every one of its subjects -- who they talk to, where they go, personality traits, etc. A global power could theoretically become omnipotent and remove any threat to its authority before it is even developed, yet legislatures are ill-prepared to recognize this pervasive threat or establish controls to regulate it. There will be catastrophic consequences if regulations don't catch up with technology, and soon.

Does this mean that the development of information technologies should be inhibited? Absolutely not. In fact, focusing on privacy concerns is a mistake when weighing the value of an information technology. We often perceive information control as the antithesis of privacy, but the most intrusive information system is (counterintuitively) the most private.

True information governance both allows us to limit who has access to information and enables transparency in an agency's behavior. It is only when an enterprise has clearly defined methods of classifying and storing information that they can truly be held accountable for their actions, an impossibility when information lies in chaos.

A Pivotal Juncture

We're now at a pivotal juncture in the development of analytics related to human behavior, and how we decide to wield this power will inevitably shape our future for better or worse. Between positive uses such as creating effective employee training and negative uses such as tracking political activists, this new breed of analytics is by far the most powerful information technology we've ever created. By harnessing our patterns of behavior it cuts to the core of what makes us who we are.

Like the discovery of fire, we can use it for good or bad, but the one thing we can't do is ignore it. It's too late to put the genie back in the bottle.