TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

Why the Key to AI Success is a Tidy Data House

Strong data management is critical to predictive and AI technology.

• By Lyndsay Wise
• March 8, 2019

Despite all the talk about artificial intelligence (AI), adoption has yet to reach its pinnacle. Many organizations are taking smart steps to implement new technologies. Instead of buying into the hype, they ask critical questions to garner the strongest ROI, resulting in a delay in broad adoption.

For Further Reading: How to Sell AI to Your Enterprise AI for BI: Better or Faster Decisions? Data Quality Evolution with Big Data and Machine Learning

Unfortunately, this is standard within the market. Organizations tend to struggle to get new applications of technology off the ground. For example, security considerations kept many organizations from adopting cloud technology, and with business intelligence (BI) in general, most adoptions follow similar paths, as companies create solutions but struggle to gain value from their endeavors.

Strategic organizations have realized strong data management is a core foundation for predictive and AI technology and are therefore focusing on getting their data house in order.

The Past

Over the last few years, the focus has been on dashboards and data visualizations. Data scientists and analysts created numerous views of the world and ways to gain insights, but organizations have struggled to manage and analyze all available data. With a multitude of sources -- from internal and external customers, consumers, partners, and suppliers -- organizations have found it difficult to create a single view of the truth. AI has the potential to support stronger data management initiatives and address a human's limited ability to accurately analyze and spot trends in the mass of data that now flows through the modern enterprise.

Early adopters of AI and machine learning (ML) must understand the underlying requirements to ensure project success for all implementations -- not just those aimed at improving internal data initiatives. Organizations look to build AI models but have not always aligned these goals with strong data management or the complexities required to create strong AI outputs. They need to understand potential biases in their data and whether they have enough data to provide valid and reliable outcomes.

Taking full advantage of AI and ML requires an understanding of the data, where it resides, what related data is required, and, finally, what initial business questions exist.

What You Should Be Doing Now

Data management is central to the emerging technologies puzzle. To this point, most organizations have faced one or more data quality problems, but the amount of data now flowing into the enterprise magnifies the issue and increases your need for a solution because as more processes are automated, inaccurate data becomes exponentially more damaging. Organizations must start by identifying what has been done to manage data previously, where it is today, where it needs to go, and how to get there, which includes developing a strong data quality framework that can maintain continuous data quality as needs expand.

For some, this means improving processes and integrating data one department at a time until the entire organization is unified. Others involve key stakeholders from the beginning -- identifying business and process challenges, identifying which groups they touch, how data is leveraged and needs to be leveraged, and how it flows through the organization. The "start small" approach may work for some organizations but comes with its fair share of challenges as companies scale their data management approach.

Manual key entry, third-party sources, and organizational silos may lead to inaccurate or unmatched data, potentially affecting how each department shares, manages, and stores its information. Because groups may have unique ways of containing and identifying data, some may find it simplest to put data in a central location with limited rules, ultimately making it more difficult for other teams to identify how the data interrelates and where the value lies. This is why it's incredibly important to have key stakeholders on board from the beginning to provide insight into how data interrelates and how it can be used across the organization.

With input from the appropriate parties, data can be stored so it can be used to solve business challenges but is not separated from people and processes. Individuals with business titles may not get their hands dirty in the nitty gritty of the data collection and analysis but it's crucial to have them involved in the process so the resulting insights provide organizational value and the flexibility needed for differing output requirements.

The Future

Emerging technologies put data front and center, forcing organizations to prioritize data management. In the past, AI was mostly hype and not part of most organizations' environments. Now, many are beginning to see the value. Every organization needs to be aware that although it may want to apply predictive models or leverage IoT analytics, many technical and business requirements must be met first. Sometimes the hype of new trends creates the perception that actual adoption is an automatic extension of current use. However, the reality for many organizations is that taking advantage of these emerging technologies requires a level of business intelligence maturity and the right infrastructure.

To take advantage of AI and ML, your organization must make sure it has all of the following:

• A mature BI environment and the skillsets to match. The adage of crawling before walking and learning to walk before running is a good way to describe the learning curve required for AI model creation.


• Data volumes that AI can learn from. Valid outcomes, without potential bias, require data volumes that support teaching the system.


• Complex questions with incomplete answers. Selecting the right models requires the intelligence of AI that is not forthcoming in traditional analytics.

AI, ML, and predictive analytics will remain front and center as they become increasingly important for efficiency and to remain competitive. Organizations that focus on building a strong foundation will reap more value from their investments in the years ahead. That starts with tidying your data drawer.