10 Rules for Real-Time Data Integration
Real-time operation is a common business requirement, and real-time data integration can enable it.
- By Philip Russom, Ph.D.
- December 11, 2012
According to TDWI’s 2011 Next Generation Data Integration survey, the types of data integration functionality that are seeing the hottest adoption and growth today are those forms that involve real-time operation. This includes DI functions that inherently run in real time, such as replication, federation, and event processing. However, ETL and variations of it are also adapting to real-time requirements by executing intraday batches to augment the usual overnight batch processing, plus integrating with message and service buses.
Organizations embracing real-time data integration (RT DI) functionality should develop policies for why and how they’ll use RT DI. The following list of rules can guide the adoption of RT DI.
Ten Rules for Adopting and Using RT DI
Rule #1: Enable fast-paced business practices with RT DI. DI’s ability to operate in real time makes it a good technology choice for real-time data warehousing and operational BI. As business management practices accelerate into real-time decision making based on complete views of customers and other parties, real-time data integration becomes even more useful.
Rule #2: Use RT DI only when a business requirement demands it. Real-time operation is like any functionality you contemplate. Don’t develop and deploy it unless there is truly a business need for it.
Rule #3: Know RT DI speeds and deliver data at the “right time.” Information and the business entities it represents have varying requirements for data freshness and delivery speed. Hence, you should study each situation to determine the “right time” required. Luckily, modern DI platforms support multiple technologies that can be configured multiple ways to adapt to the speed and frequency needed.
Rule #4: Expect to do RT DI with more federation, services, and events. According to TDWI surveys, these are the most aggressively adopted DI techniques of recent years. Your peers working in other organizations are adopting these; perhaps you should, too.
Rule #5: Use DI techniques that are conducive to RT. In addition to the DI techniques mentioned in Rule #4, others conducive to RT operation include replication, change data capture, and microbatch ETL.
Rule #6: Do not scrimp on data prep for the sake of speed. Let’s be honest: fetching, preparing, and presenting data takes time for processing. Even if it slows down the total response time, include as much validation, standardization, and quality processing you can. Otherwise, you may be presenting bad data, albeit very quickly.
Rule #7: Mix RT operational data with historic DW data. Data extracted frequently from operational applications can tell you what happened in the last few minutes or hours. To provide a longer-term context, present RT data along with historic time-series data from a data warehouse or similar database.
Rule #8: Develop one DI solution; deploy in many speeds, on many interfaces. It’s often the case that you have a single processing task (such as a data extract, standardization, or transformation) that needs to operate at more than one “right time.” Try to develop one task (or method, routine, object, or whatever your tool calls it), then determine the right time at deployment time or even automatically during execution.
Rule #9: Develop policies to govern the use of RT functions and standards. In one direction, this means collecting and examining business requirements carefully, to be sure the business can truly use real-time operation for some kind of advantage. In another direction, you should have development standards that identify preferred technologies and interfaces for common RT DI configurations.
Rule #10: Evangelize operational BI as RT DI’s killer app. For years at TDWI, we’ve seen the growing popularity of operational business intelligence as one of the first use cases for RT DI adopted by user organizations. The business value of operational BI is readily clear to most people, so it makes a useful example, when talking up RT DI as a key enabler for fast-paced business practices.
For more information, replay the 2012 TDWI Webinar, Real-Time Data Integration: An Infrastructure That Enables Fast-Paced Business available online at http://bit.ly/RealTimeDI.