Building the Real-Time Enterprise
Excerpt from the Full Report by Colin White, President, BI Research
The TDWI report was sponsored by Ascential Software Corporation, Business Objects, Celequest Corporation, DataMirror, Informatica Corporation, and Teradata, a division of NCR
WHAT DO WE MEAN BY REAL-TIME BI?
The term real time has become synonymous with an organization's ability to become more agile, and thus more responsive to changing business circumstances. Although this agility undoubtedly can provide business benefits, there is still universal confusion about what real-time BI means from both a business and technology perspective.
In an ideal world, organizations would react instantly to business needs. In reality, however, this is not possible. Accurate business intelligence is required for effective decision making. And it will always take a certain amount of time to collect and deliver this information to business users, and for users to act on this information. The delay, or latency, between a business event occurring and action being taken, depends on business user reaction time, and on the technologies used to deploy real-time processing BI applications.
Right Time Not Real Time. When developing real-time applications, organizations need to consider how responsive each business process must be to satisfy a specific business goal. From a business perspective, it's more useful to think in terms of "right time" rather than "real time."
Depending on business needs, the action time requirements in real-time environments may vary from a few seconds to several hours. A practical definition, therefore, for the meaning of real time is "the ability for an organization to react to business needs and changing business circumstances within a single day." Many organizations we spoke to agreed with the intra-day definition of real time because this threshold is important from both a business and a technology perspective.
Business Benefits. For the business, the ability to take intra-day action can have a significant positive impact on both business finances and marketplace perception because this capability enables an organization to use business information to drive its daily business operations efficiently. Although the ROI realized by reducing action time to a specific level is the main factor in justifying real-time projects, this ROI must be balanced against the incremental IT costs of achieving the reduction.
IT Requirements. For the IT group, the technology infrastructure required to deliver business information and support business decision making and actions within a one-day period is more demanding, and often quite different, from that used to deploy applications with less rigorous action time requirements. A wide range of technologies and products support real-time processing, but this is a new and rapidly evolving field, and it is important that real-time infrastructure be designed to weather this evolution. It is vital that this infrastructure provide scalability, performance, and availability.
MORE THAN JUST TECHNOLOGY
Real-time processing involves more than just technology. It must be recognized that the business ROI achieved by reducing business reaction times will also depend on the ability of the organization to modify its business practices to take advantage of the improved responsiveness in the IT system.
Our research shows that success in the real-time environment requires the IT organization, from both an organizational and service level viewpoint, to eliminate the distinction between operational and decision support processing. Removing the barrier between operational and decision support IT groups eliminates "turf wars" and gives the IT organization access to both types of skills on real-time projects.
Real-time processing offers significant business benefits, and there are many examples of successful real-time projects. It must be realized, however, that this is still a leading-edge technology. And organizations must track the evolution of the real-time processing market and update real-time processing strategies and frameworks accordingly. It will also be vital to educate both business and technical staff about the benefits, technologies, and pitfalls involved in deploying a real-time enterprise.
THE BUSINESS CASE FOR REAL-TIME BI
Balancing Business Benefits and IT Costs. Real-time BI enables organizations to react more quickly to changing business circumstances and requirements, but the business benefits and ROI must be balanced against the IT costs of making the BI system more responsive. To understand this trade-off we need to examine the components of a real-time BI system, and how they affect the business decision-making process.
"There will always be delay between a business event occurring and appropriate business action being taken," says Richard Hackathorn, President of Bolder Technology. "In a business intelligence environment, this delay has three components: data latency, analysis latency, and decision latency. Data latency is the time between the business event occurring and when the data about the event is ready for analysis in the data warehouse. Analysis latency is the time required to analyze the data and send the results to the user. Decision latency is the time required for the user to understand the information and take the appropriate action." This concept is depicted in Illustration 2.1.
Illustration 2.1. Latency in business decision making. Based on a concept developed by Richard Hackathorn, President of Bolder technology. Reproduced by permission.
The sum of the three latency periods is the action time or action distance, and the objective of a real-time BI system is to reduce the action time required to respond to a business event. Data latency and analysis latency can be reduced by technology innovation, but reducing decision latency depends on the business user. Providing the user with better information will of course help reduce decision latency. Rather than simply telling users a problem has occurred, for example, it is of more value to also provide detailed analyses and suggest actions. Another approach to reducing decision latency is to replace manual user decision making with rules-driven BI processes that automatically take action on behalf of the user.
A real-time BI system has two main components: real-time data integration and real-time decision making (see Illustration 2.2). The objective of the real-time data integration component is to capture business events from operational systems and integrate them into a low-latency store. This component supports the real-time data-on-demand type of real-time processing. The real-time decision making component, on the other hand, supports real-time performance management and real-time predictive analysis.
Illustration 2.2. Real-time BI processing components.
In our survey, 50 percent of the 846 respondents either had deployed, or were planning to deploy, real-time BI projects. Of these, 21 percent were doing real-time data integration, 15 percent were doing real-time decision making, and 57 percent were doing both. Follow-up interviews indicated that many of the companies doing just real-time data integration were simply loading their data warehouses faster to alleviate performance and operational batch window problems. Organizations doing just real-time decision making, on the other hand, were usually using data in operational systems, rather than in a low-latency store, for rapid decision making. Overall, then, about a quarter of companies either had implemented, or were planning to implement, a full real-time BI system.
The survey results also demonstrate that some people still think real-time BI is about loading data into a data warehouse faster. Although this helps reduce data latency, it does nothing to reduce analysis or decision latency.
"A real-time BI system is not about keeping data current in the data warehouse and then telling the user to access it. It's about delivering the right information at the right time to the right business users to enable them to react rapidly to solve business problems," says Phillip Gollhofer, Manager of Business Intelligence at BNSF.
Although the ROI realized by reducing the action time to a specific level is the main factor in justifying real-time projects, this ROI must be balanced against the increased IT costs of achieving the reduction. The IT costs will depend on things like the amount and type of business information that has to be accessed and analyzed, and the BI system performance and availability requirements of business users. As shown in Illustration 2.3, the smaller the action time required, the more expensive the IT solution.
Illustration 2.3. Real-time BI: Action time versus IT costs.
It should also be recognized that the business ROI achieved by reducing action time depends on the organization's ability to modify its business practices to take advantage of the improved responsiveness in the IT system. Several organizations we interviewed said this was an important success factor in realizing the benefits of a real-time BI system. There is a point beyond which reducing the action time any further has no business value because the increased responsiveness cannot be exploited by the business. This is shown as the exploitation threshold in Illustration 2.3.
"Based on discussions with business users, we chose to update the real-time component of the data warehouse every two hours," says BNSF's Gollhofer. "This is a trade-off between making rapid decisions, keeping the data stable long enough to enable it to be analyzed by users, and IT costs."
"People should not think in terms of real time, but should consider how responsive each business process needs to be to satisfy a specific business goal," says Stephen Brobst, CTO of Teradata. "Organizations should think in terms of right time, rather than real time." This viewpoint is shared by David Randall, Pre-sales Director at DataMirror, who states that, "Right time is the appropriate term to use because each business process has its own unique requirement about how fast the organization needs to react to a particular business situation."
APPLYING REAL-TIME BI TO BUSINESS PROBLEMS
In our survey we asked companies to identify the business area where they were using, or planning to use, real-time BI. As the results in Illustration 2.4 show, sales, marketing, customer service, and finance/accounting are the four main areas of focus.
| | | | | | | | |
| Other |  |
| Not sure | |
| Supply chain management | |
| Procurement | |
| Manufacturing | |
| Finance/accounting | |
| Human resources | |
| Customer service | |
| Order management | |
| Sales | |
| Marketing | |
| | 0% | 10% | 20% | 30% | 40% | 50% | 60% |
Illustration 2.4. What business areas do or will your real-time project support? Multi-choice question, based on 417 respondents.
Based on discussions with many people in the industry, we put together a list of potential business uses for real-time BI, and asked our survey respondents to use the list to identify real-time applications that were being implemented in their organizations. The results (see Illustration 2.5) showed that a single view of the customer, financial reporting, and sales forecasting and pipeline monitoring were by far the most common uses of real-time BI. Some 10 percent of respondents added several other real-time BI applications to the mix.
| | | | | | | | |
| Other | |
| Patriot Act compliance | |
| Product recalls | |
| Distribution chain optimization | |
| Real-time inventory analysis | |
| Manufacturing processing and plant maintenance | |
| Just-in-time inventory | |
| Quality control and monitoring | |
| Risk management | |
| Financial reporting | |
| Program trading (stocks) | |
| Customer portfolio management | |
| Automated loan/credit card processing | |
| Claims processing | |
| Fraud detection | |
| Order management | |
| Sales forecasting and pipeline monitoring | |
| Product pricing and promotions | |
| Real-time marketing | |
| Single view of the customer | |
| | 0% | 10% | 20% | 30% | 40% | 50% | 60% |
Illustration 2.5. Which real-time BI applications have you implemented or do you plan to implement? Multi-choice question, based on 417 respondents.
Illustration 2.6. Examples of real-time BI applications.