Data Integration and Machine Learning: A Natural Synergy
In the age of machine learning and artificial intelligence, data integration plays a key role in the digital transformation of any enterprise in any industry.
- By Arun Patnaik
- August 18, 2017
The Need for Analytics
In the early days, IT practitioners built reports and dashboards to present data business users could consume and interpret. Over time, these dashboards needed to be more timely and interactive. Business intelligence systems, backed by a data warehouse, were built to allow such interactive analysis. Data was extracted from operational systems, cleansed, and transformed to be consistent and suitable for analytics, and data warehouses were populated with this clean data. The process was referred to as data integration, and it began to play a vital role in transforming data into insight.
Today's business environment demands advanced, near-real-time analytics to enable automated business processes and to let business users gain valuable customer insight. Enterprises are harnessing advances in computer technology to achieve this goal. Data integration plays an important part.
More Data, More Problems
True to Moore's law, computational capability has grown exponentially over the years. With increasing computational capability, mining a large amount of data for insights is becoming easier. The ability to utilize vast amounts of data has increased awareness of the data's value.
There's now more data to analyze than ever before. The recent growth in social media and machine data is producing exponentially larger amounts of data. Quantity isn't the only issue. The variety of data used for analytics has grown to include semistructured and unstructured data, including text, image, and voice data. Overall there is huge growth in both the volume and variety of data captured in any form, requiring new capabilities to corral and tame this data.
Growth of Machine Learning
As data volume and variety have increased, so, too, has the acceptance of machine learning. The exponential growth in computational power has also improved the effectiveness of machine learning-based algorithms to the point that machine learning is now a viable tool for analytics.
Compared to traditional algorithms (where a human explicitly codes an algorithm), machine learning-based approaches are more elegant because they can implicitly learn based on actual training data. Furthermore, machine learning-based approaches are more flexible because they can adapt to changes over time by learning from feedback. Thus, effective machine learning implementations often replace explicit programing. For example, machine learning-based search-result ranking algorithms have subsumed explicitly programmed, yet successful, algorithms such as the "page rank" algorithm. We are beginning to trust machines or automated software even though we don't fully comprehend them, as evidenced by the success of route planning recommendation systems such as Waze and Google Maps.
As with any new technology, there is hype about machine learning. The technology is being adopted in a variety of projects across the enterprise in analytics or other core business functions (such as pricing, forecasting, and planning).
The Machine Learning/Data Integration Connection
For machine learning to be effective and analysis to be comprehensive, enterprises must utilize data from the greatest possible variety of sources. A machine learning algorithm is only as good as the data used to train it. Although there is an abundance of enterprise data, much of it is still not easy to find or use. This type of data is called dark data. Enterprises are struggling to throw light on this dark data and make use of it.
Data integration systems have risen to this challenge, resulting in an emergence of several data catalogues and data lake management products. These products aspire to be the "Google for enterprise data" and offer a simple search-based interface to find and explore all the data in an enterprise while honoring existing access control mechanisms.
Data integration systems are increasingly looking to use machine learning-based approaches for finding and highlighting the islands of useful data in the vast ocean of dark data (and thus improve analytics). Metadata is gaining a stronger emphasis and is being captured explicitly or inferred with help of machine learning. Some examples are the use of machine learning in the inference of schema, data distribution, and common value patterns.
Machine learning algorithms are employed on metadata, social context, and operational characteristics to identify accurate, clean, and relevant data for various analytics exercises. For example, in the context of data catalogues, clustering algorithms can be used to group similar data sets, and then collaborative filtering algorithms can be used to recommend the more useful ones among them in each context.
Similarly, in the context of data protection, classification algorithms can be used to automatically detect sensitive data and protect them using an appropriate scheme in a policy driven manner. These are only a couple of examples of how data integration systems are applying machine learning to improve analytics. Every dimension of data management is evolving to make space for applying machine learning to improve the whole process.
Machine learning and data integration making each other more effective is a true example of a symbiotic system. This is just the beginning of what promises to be an exciting journey.
About the Author
Arun Patnaik is a Distinguished Engineer at Informatica where he is responsible for driving the Informatica technology architecture. You can reach the author at [email protected].