Chasing the Real-Time BI Dream
It is time for your organization to put "achieving at or near real-time views of data" higher on its strategic priority list?
- By David Stodder
- October 21, 2014
Business intelligence has traditionally been thought of as limited to historical views of data. Yet, driving out data latency and shortening query response times have been key goals throughout the entire evolution of BI technology and practices. Users have been the driving force behind this evolution. Since the days of executive information systems (EIS), business executives and managers have dreamed of having something closer to real-time data in their applications -- and have been loudly disappointed about not getting it. Although just getting access to data through an EIS or BI application was a big step, most users knew that the older the data, the more its potential value to the business was lost.
BI applications by themselves cannot deliver real-time or near-real-time data; it takes orchestration of an entire stack of technology, not to mention good data models, well-written queries, and intelligent optimization to ensure that users get the right results as quickly as possible. Sometimes organizations have to wait for an upgrade to an underlying system to seize the opportunity to realize users' near-real-time BI dreams. When such an opportunity arises, BI planners have to be ready to make their case for closing the data latency and query response gap.
This point was brought home to me in my research for the latest TDWI Best Practices Report, Real-Time Data, BI, and Analytics, which I co-wrote with my colleagues Philip Russom and Fern Halper. We interviewed Brad Sheridan, manager of BI at the Howard Hughes Medical Institute (HHMI), a leading nonprofit science philanthropy that funds researchers at a variety of host institutions around the U.S.
Sheridan said that the Institute depends on an aging ERP system for data updates, which occur overnight in batch. The Institute's data warehouse is therefore also limited to overnight updates, meaning that the data for BI is always a day behind. "It's frustrating to do something 20 minutes before leaving the office knowing you can't pick up the results of your query until the next morning," he said.
The good news is that HHMI is in the process of evaluating the replacement of its ERP system, which will create an opportunity to make big changes to the BI platform, data warehouse, and data marts as well. Intra-day refreshes are in the plan as an objective of upgrades to the Institute's data warehouse and BI reports. Sheridan said that HHMI is planning for a new data warehouse running on a columnar database; this will be focused on supporting visual data discovery and analytics of financial data for the Institute's scientists as well as finance and administrative personnel.
New Business Opportunities with Real-Time Data
Another company we interviewed for the report was Stanley Healthcare, a division of Stanley Black and Decker. When one thinks of Stanley, screwdrivers, hammers, and other home improvement tools usually come to mind. However, in 2012 the company bought AeroScout, which develops real-time location services (RTLS) technology; this is now the core solution provided by Stanley Healthcare. The RTLS system "constantly sends data from tags and other sensors into the database," said Joel Cook, director of healthcare solutions at Stanley Healthcare. Viewing the data through its software application MobileView, users can be alerted to changes and interact with the data visually. The RTLS system has a complex event processing engine that manages real-time alerting to users and systems.
Stanley Healthcare's solution has been applied to a variety of industries, but it has become well known in the healthcare field for use in infant tracking, environmental monitoring, asset management, and patient and staff safety monitoring. "Hospitals, under pressure to improve efficiency without sacrificing patient care, are excited about what they could do with [MobileView's real-time event] knowledge," said Lauran Hazan, Stanley Healthcare's director of software product management.
Faster BI is Usually Better BI
Timelier data and quicker query response are necessary for meeting increasingly dynamic business needs. New technologies for capturing and analyzing events in near real time are becoming less exotic and more part of the mainstream. With options such as external data services, analytics in the cloud, and deployment of business-driven, "shadow" IT systems in front of them, users are less willing to wait for IT to deliver solutions that close the latency gap. Getting timelier data and quicker query response are major reasons why business leadership will seek to deploy systems outside of IT's direction.
Most users of BI applications are not happy with old data; they can envision something better. Organizations should evaluate whether they need to move achieving at or near real-time views of data up higher on their strategic priority lists. Certainly, when major underlying systems or applications are being overhauled, organizations should upgrade users' ability to access, analyze, and visualize more timely data.