Freedom from Batch Windows Using Real-Time Data Integration
By Irem Radzik, Director of Product Marketing, Data Integration, Oracle
We expect freedom when we invest in technology solutions. Freedom means having the power, choice, and flexibility to improve the way we do business—raising the efficiency of IT and delivering results to the business faster. If we look at data warehousing specifically, we see it as a critical technology solution that provides a complete picture to steer the business in the right direction. Although data warehouses are designed to give us the freedom to access complete and reliable information, the underlying data integration architecture and the type of solution used can lead to significant constraints on how we manage our critical production systems.
One of the most common constraints is the available time window for batch extract processing on the source systems. Because batch extracts are resource intensive for the production environment, they are completed in off-business hours, typically overnight. The problem occurs when off-business hours start to disappear. For businesses that have global operations or support customers 24/7, their critical systems do not have adequate off-business hours to allow batch extracting. This problem becomes more pressing when there is an ETL process interruption or failure during the nightly batch window that prevents the data extract and load from completing in time. In some cases, a day’s worth of data may be missing the next day in the data warehouse due to a small hiccup.
IT organizations try to remedy this batch window problem by speeding up ETL processes to complete within the allotted time window. However, this is a temporary solution that does not address the core of the problem—the overhead imposed on source systems. It does not address the risks involved with process interruptions and related data inconsistency issues, either.
How do you liberate your critical production systems from this batch window constraint? As our case study (previous page) illustrates, a low-impact, real-time data integration solution has proven to be effective in freeing up source systems from nightly batch extract processes. When the extract component uses a nonintrusive method, such as reading database transaction logs to capture only the changed data, it does not burden source systems. Hence, data extract can happen at any time of the day, while all users are online.
Oracle GoldenGate offers log-based changed data capture, routing, transformation, and delivery to support data warehousing solutions with a nonintrusive, real-time data integration solution. It integrates with Oracle Data Integrator, which uses ELT architecture and provides high-performance data loading and in-database transformations for the data warehouse in near real time. By moving only the changed data throughout the day, Oracle’s solution removes batch window dependency. This solution architecture also allows more efficient use of network bandwidth resources.
In addition, Oracle GoldenGate is designed for reliable data delivery. Captured changed data is immediately stored in trail files that can reside in source and target servers outside the databases. The trail files can be used for complete data recovery in case there is an outage in the data warehouse during the loading process. Finally, an obvious but critical benefit of using a real-time data integration solution is feeding up-to-date data to the data warehouse, which helps improve business insight.
Many GoldenGate customers, including the SunGard example provided on the previous page, see tangible benefits from using continuous log-based changed data capture and delivery technology in their data warehousing solutions. In addition to high ROI, this technology gives the OLTP users the freedom to access source systems when they need to, and allows the IT staff to sleep at night with peace of mind.
This article originally appeared in the issue of .