The State of Operational Data Warehousing
By Philip Russom
Status of OpDW Implementations
This discussion of operational data warehousing and related matters may make you wonder how many user organizations are actually doing it. To that end, this report’s survey asked respondents: “Does your organization practice some form of operational data warehousing today?” (See Figure 3.)
Two-thirds of organizations practice some form of OpDW today. Sixty-six percent of survey respondents answered “yes,” which gives OpDW users a clear majority over the 31% who answered “no.” Note that the 66% includes a wide range of technologies and practices, whether operating in real time, near real time, on demand, intraday, or latently overnight. The wide range explains why so many respondents claim they are doing OpDW today. By comparison, far fewer are pushing OpDW to the extremes of real time today, as revealed by the 17% figure cited earlier.
OpDW awareness is high. Survey respondents seem well-informed about OpDW; only 3% said they don’t know whether their organization is doing it. This suggests that users are discussing the various forms of OpDW with their peers and management; otherwise, they wouldn’t know whether it’s being used. In fact, the results of this survey in general show a high level of awareness of OpDW and its variations, more so than with TDWI surveys about other topics.
To get a rough sense of what lifecycle stages organizations are in with OpDW, the survey asked: “What’s the status of your organization’s OpDW implementation?” (See Figure 4.)
Roughly half of organizations are committed to OpDW. More than half of user organizations surveyed (56%) have made a commitment to some form of operational data warehousing, whether their solution is currently in development or already deployed. Given the wide margin of accuracy of this type of survey, the 56% committed to OpDW in Figure 4 is in the same ballpark as the 66% claiming the use of OpDW in Figure 3. Both figures corroborate that OpDW is commonly used today, but with ample room for growth in the future.
More organizations are likely to commit soon. A large percentage of organizations currently have OpDW under consideration (29%), and no doubt many of these will progress into OpDW usage. Furthermore, relatively few organizations have no plans for OpDW (15%). For these reasons (and for the reasons stated earlier in “Why Care About OpDW Now?”), TDWI Research feels confident that OpDW usage will increase.
Benefits of OpDW
OpDW can result in benefits that positively affect data, the enterprise, and the management of each. To get a sense of which benefits are more likely than others, this report’s survey asked: “Which of the following would improve in your organization if you implemented some form of OpDW?” The most likely benefits (seen at the top of Figure 5) are those most often selected by survey respondents, and the likelihood of a benefit declines as the list proceeds downward.
Business intelligence tasks are the top beneficiaries of OpDW. Almost half of respondents selected business decisions and strategies (46%) as things that would benefit from OpDW, and these are typically enabled by BI tools and techniques. Related to BI, data-driven corporate objectives (14%) and views of the business via data (13%) are also potentially benefited by OpDW.
Business operations are likely to gain from OpDW. Survey responses indicate strongly that OpDW can improve business performance and execution (39%) and operational excellence (24%). In related operational issues, OpDW can also contribute to the efficiency of business operations (33%) and perhaps even lower the cost of business operations (19%).
Data currency improves when OpDW operates in real time. As established earlier in this report, integrating operational data in real time or with intraday frequency is a common technical component of OpDW, though not required for all applications. Given that most OpDW implementations include these capabilities, it’s no surprise that survey responses identified data freshness or timeliness (35%) as an area improved by OpDW. When the currency of data improves, so does employees’ ability to be proactive (17%).
Data sharing is expanded by OpDW. With that in mind, OpDW can improve the business leverage of data assets (31%), data sharing across business units (23%), and cross-unit business processes (16%).
Improvements to most data attributes is unlikely with OpDW. Data currency and sharing aside, other attributes of data and its management ranked somewhat low in the survey. According to respondents’ perceptions, OpDW is not as likely to improve data’s quality (19%), governance (16%), architecture (15%), metadata (9%), or models (9%).
To summarize, the leading potential benefits of OpDW are improvements to business intelligence, business operations, data currency, and data sharing.
Barriers to OpDW
OpDW has its benefits, as we just saw. Yet, it also has its barriers. Again, to get a sense of which barriers are more likely than others, this report’s survey asked: “In your organization, what are the top potential barriers to implementing OpDW?” The most likely barriers (seen at the top of Figure 6) are those most often selected by survey respondents, and the likelihood of a barrier declines as the list proceeds downward.
Cost (44%) is the most likely barrier to OpDW adoption and success. This is natural, given that economies around the world are struggling to recover from one of the worst economic recessions in recent centuries. Of course, cost consciousness is still heightened, and many IT budgets are still cut, locked down, or tightly controlled. But there’s more to it than the economy.
In the recent past—say, as late as 2002 or so—implementing OpDW usually meant purchasing and implementing additional technologies for real-time operation and interoperability between operational applications and BI/DW tools. The situation today is very different. As we’ll see in the next section of this report, organizations that have kept up-to-date with BI/DW platforms and IT infrastructure already have all or most of what they need for many configurations of OpDW. Yet, the outdated perception persists that OpDW demands the acquisition of new and expensive tools.
As with most initiatives, OpDW won’t succeed without the business behind it. Perhaps that’s what survey respondents were thinking when they selected certain barriers, such as the lack of business sponsorship (31%), lack of compelling business case (22%), and lack of governance or stewardship (18%).
The weaknesses of a data warehouse platform or team can be impediments to OpDW. For many users, the sad fact is that their current data warehouse can’t handle data in real time at all (28%). TDWI’s 2009 research into next-generation data warehouse platforms showed that migrating to a platform that is inherently real time is the leading reason for replacing a data warehouse platform. Closely related barriers arise when existing BI and data integration tools are not conducive to OpDW (18%) and the current data warehouse can’t process tactical queries fast enough (12%). Some users worry about the difficulty of architecting a complex OpDW system (20%), which is exacerbated by their lack of staff experienced with OpDW (20%) and the fact that the current data warehouse is modeled for historical data only (14%).
Problems with enterprise data infrastructure can inhibit OpDW implementations. These include poor master data or metadata (19%), poor quality of data (18%), inadequate data management infrastructure (17%), and interfaces to operational applications that are too slow or feature poor (13%).
To summarize, the leading potential barriers to OpDW are cost (whether actual or perceived), the lack of business support, weaknesses in the current data warehouse platform, and problems with enterprise data infrastructure.
Philip Russom is a research director at The Data Warehousing Institute (TDWI), where he oversees many of TDWI’s research-oriented publications, services, and events. Prior to joining TDWI in 2005, Russom was an industry analyst covering BI at Forrester Research and Giga Information Group. He has also run his own business as a BI consultant and independent analyst, plus served as a contributing editor to leading data management magazines. You can reach him at firstname.lastname@example.org.
This article was excerpted from the full, 32-page report, Operational Data Warehousing: The Integration of Operational Applications and Data Warehouses. You can download this and other TDWI Research free at tdwi.org/bpreports.
The report was sponsored by EMC, HP Business Intelligence Solutions, SAP, SAS, Syncsort, and Teradata.
This article originally appeared in the issue of .