TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

How Data Preparation Can Accelerate AI

These four best practices can help your organization quickly prepare data for analytics and fast-track its algorithms into production.

• By Kim Kaluba
• March 13, 2020

It’s rare to meet a leader who isn’t excited by the potential of AI and analytics to drive their organization’s digital transformation, growth strategies, and operational efficiencies. The success of an AI-based technology revolution -- or even the build of a very simple algorithm -- ultimately lies in the health of the data.

For Further Reading: AI and BI Projects Are Bogged Down With Data Preparation Tasks How to Cut Data Preparation Time for Visualization Tools Accessible Data Preparation: 6 Data Quality Tips

Only after data completes various test drives (or emerges from data preparation steps) does it finally become qualified for analytics and AI model development. However, in survey after survey organizations continue to report problems with accessing, preparing, cleansing, and managing data, ultimately stalling the development of trustworthy and transparent analytical models.

Idling in Neutral on the Data Highway

Why are organizations missing the exit for transformative business insights from AI and instead sitting in traffic on the data management highway? There are three key reasons:

• Analytics teams are spending too much time preparing data. Data scientists spend between 60 and 80 percent of their time on data prep, leaving insufficient time to interact with data, run advanced analytics, train and evaluate models, and deploy models to production. The problem is that there is no straightforward way for business analysts, citizen data scientists, and other non-IT roles to move and transform data for analytics. Data assets cannot be utilized because business users lack resources that do not require coding skills or extensive data integration know-how, affecting data latency and resulting in data that isn’t fit for analytics.


• Business departments need access to data in a timelier manner. In a competitive market, business departments can no longer rely on traditional ETL methods to keep up with real-time demands. Business analysts and data scientists alike are spending an excessive amount of time waiting on IT to provide data, ultimately hampering the organization from reacting effectively to changing market conditions.


• The increased number of locations where data resides makes accessing the data and finding the right data challenging. A vast data environment can limit data transparency. Very often business users do not even know what data assets are available because there is no up-to-date documentation or search interface available for them to find that information. In addition, immense data environments have forced company data usage to be fragmented. The result is that decisions are made in silos and conflicting information on reports undermines effective decision making.

Putting the Pedal to the Data Metal

Given these roadblocks, what are the best approaches companies can take to accelerate preparing data for analytics and fast-track their algorithms into production?

Best Practice #1: Automate and augment data processes to expedite data prep

Allow users to take advantage of AI and machine learning to scan data and intelligently make transformation suggestions while enabling users to accept the suggestions and complete the transformations with a simple click of a button. Some examples of automated suggestions for data are gender identification, standardization, matching, and deduplication.

Best Practice #2: Use self-service data preparation tools that do not require advanced coding skills or reliance on IT

You don’t want users to spend time performing advanced or complex coding because it lengthens the time it takes to get to analytics and insights. Whether or not you accelerate your self-service data preparation using AI, your data preparation tool should provide profiling, browsing, and filtering tools as well as data preparation features, which include structuring, transforming, and formatting data.

Best Practice #3: Develop collaborative workflows within the self-service environment to eliminate silos

Provide the ability to share plans, work, and insights among teams and individuals to improve reusability and shareability of vetted data pipelines and to expedite data prep.

Best Practice #4: Use the cloud

Businesses need to securely move high-volume data from on-premises data stores to the cloud and vice versa. They need to securely read and write on-premises data to the cloud and use that information for analytics and decision making.

A Final Word

Devoting time to data preparation is important, considering that the better the data that goes into building the analytical model, the better the output. Your enterprise must ensure users that the data used in their analysis is properly clean, enriched, and formatted. Data preparation provides that trust.

Your digital transformation strategy does not need to be stalled in a never-ending work zone of data preparation. Instead, find the smart detours that safely lead to AI deployment and don’t limit your speed in arriving at insights.