LESSONS - Implementing Real-Time Data Integration Solutions
By Jennifer St. Louis, Manager of Marketing, DataMirror, IBM Software Group
With customer loyalty waning and significant competitive pressures on organizations in every industry, companies need to find innovative ways to win business so they can stay ahead of the competition.
Delivering trusted information wherever and whenever needed, in line and in context, to specific people, applications, and processes, allows organizations to gain a competitive advantage in their marketplaces. Companies need to respond more quickly, gain operational efficiencies, and deliver superior customer service by keeping a real-time pulse on their business. Although most organizations already possess the information required to do this, they are often unable to get it when and where they need it.
A real-time data integration strategy ensures accurate data flows across the enterprise so that organizations can make quick decisions on pricing, shelving, service, and product mix, based on the latest information. Managers can analyze critical business data throughout the day to target their marketing efforts, improve up-selling and cross-selling strategies, and better service their customers, no matter what business they’re in.
A solid, real-time data integration strategy can give organizations the information they need to get real insight into their business and create a competitive advantage.
There are four key technical challenges organizations face when implementing a real-time data integration strategy:
1. There is too much information, and users are unable to determine which information is important.
Organizations have great volumes of data and know they can benefit from this information; but many do not know how to pull it together and make it useful to the business. Common examples include not using demand signals to drive supply chains in the retail space or not enabling customers with the most upto- date banking information in the financials vertical. Companies that do consolidate and transform data can optimize their supply chains, get better results, and improve customer satisfaction.
2. Data contains multiple versions of the truth.
Different systems store different pieces of information on core entities, such as product, customer, or partner. This information quickly gets out of synch, making it difficult to get a single view. Commonly, the result is that organizations have problems in managing customer, product, and partner interactions. Also, regulatory compliance is inhibited by poor visibility into the information needed.
3. Customers face issues associated with a lack of understanding of their data.
Without insight into the quality of their data or where the right information resides, organizations may have incomplete and outof- date or inaccurate data that can quickly proliferate throughout the organization, causing rework, lost time, and lack of trust from end users.
4. Organizations must deal with a lack of agility.
Organizations are unable to take advantage of opportunities for innovation because their systems are too inflexible. They cannot adjust quickly enough to take advantage of new opportunities, and systems are expensive to maintain. Costs tend to escalate due to inflexible systems and manual efforts required to meet the changing needs of the business.
For these reasons, a solid, real-time data integration strategy can give organizations the information they need to get real insight into their business and create a competitive advantage. Yet with growing volumes of data residing in multiple applications, it hasn’t always been easy to get trusted information when you need it most.
Implementing a data integration solution should always start with examining your business requirements and deciding where and when you need information delivered.
Implementing a real-time data integration solution
The majority of organizations use ETL tools as their primary means for data integration. They use these tools, or homegrown data integration processes, to extract data in bulk from their production systems and load it into other systems, including data warehouses. The main strengths of these tools are that they extract data from many different applications, perform complex transformations and data quality on that data, and then load large volumes of data into data warehouses. But when it comes to extracting data from production systems or providing real-time visibility into operational systems, ETL tools sometimes need to leverage real-time capabilities. Increasingly, it is becoming necessary for organizations to conduct business around the clock. As more business is done across time zones and over the Web, more organizations are faced with the problem of shrinking batch windows—making it more difficult for traditional ETL tools to extract data in the short time available.These tools were not built for keeping multiple applications in synch with real-time data feeds.
Implementing a data integration solution should always start with examining your business requirements and deciding where and when you need information delivered. IT then needs to look at where data resides throughout the organization.
This step involves meeting with all interested parties—LOB managers, end users, application programmers, database administrators, etc. During this process, collect all information about the existing IT environment, including what systems the data resided in and where and when the information needs to be accessed. Perform analysis to determine the application transaction volumes to assist in optimizing the implementation.
In the next step—analysis and design—IT must define the project scope and architect the optimum replication scenario for the environment. At this point, complete the analysis and documentation of the project and environment. Recommend replication architecture and solutions, and determine timeframes.
In architecting a solution, keep in mind the effect it will have on source applications. Change data capture solutions can be used to complement or leverage existing ETL processes by providing real-time data flows that capture transactions directly from database logs and send those transactions to existing processes. Take care not to add additional workload on production applications and networks.
After the solution is determined and implemented, completely check the environment to ensure the data integrity and health of existing production applications. The impact of implementing real-time data integration must not negatively affect users of production systems. During this step, review the complex business rules to ensure that they meet the needs of the user environment. Repeat this step to ensure that the solution continues to meet the needs of the organization.
Once a real-time data integration strategy is implemented, it can ensure that accurate data flows across the enterprise, allowing organizations to synchronize information across all their customer touch points. Organizations can gain an immediate single, complete, 360-degree view of customers so they can target their marketing efforts and improve their cross-selling strategies.
Today, only enterprises that can effectively manage, integrate, distribute, and utilize their data assets will survive and prosper. An increasing number realize that without continuous, real-time data integration, they cannot achieve the required visibility into their existing systems. Taking real-time data flows to the next level will enable organizations to sense and respond to data changes in real time. With this ability, they can proactively service their customers and support initiatives, including dynamic warehousing, master data management, SOA, migration/consolidation, and e-business.