Executive Summary: Real-Time Data, BI, and Analytics
- By Philip Russom, Ph.D., David Stodder, Fern Halper, Ph.D.
- September 24, 2014
User organizations continue to push their applications for business intelligence (BI), analytics, and
data management closer to real-time operation. This is because fresh information can support fast-paced,
time-sensitive business processes, such as operational BI, real-time management dashboards,
just-in-time inventory, high-yield manufacturing, facility monitoring, call center information
delivery, self-service information portals, recommendations in e-commerce, and so on.
According to this report’s survey, the leading benefits of implementing real-time technologies for data,
BI, and analytics include better actions based on BI and analytic information, improved customer
service and customer experience, and automated decisions made by software, not people. The barriers
to achieving such benefits include cost, difficulties in designing real-time systems, the state of data
management infrastructure, and inadequate staffing or skills.
The term real time has become an umbrella concept encompassing multiple time frames, speeds, and
execution frequencies. BI and analytic practices are progressively rarer as we come closer to true real
time (milliseconds and nanoseconds). For example, most reports and analyses today operate on data
that’s refreshed daily on a 24-hour cycle. Even so, BI and analytics based on hourly cycles is well
established, and so-called near-real-time cycles (seconds and minutes) are fairly common.
A variety of technologies are in use today, enabling real-time and near-real-time data, BI, and
analytics. Common ones include data federation, replication, data sync, message buses, and
microbatches. A number of functions designed for high performance are increasingly applied to near-real-time uses, namely changed data capture, columnar databases, in-database analytics, in-memory
processing, and solid state drives. Organizations in need of true real time are ramping up the use of
complex event processing (CEP) and continuous stream mining.
The proliferation of operational BI is the leading driver for real-time usage in BI. Most operational BI
users interact with near-real-time data via management dashboards and other types of reports.
However, users are aggressively adopting visual data discovery tools because of their ease of use,
strong self-service functions (which empower many user types), and the tools’ ability to process large
data sets with near-real-time performance.
As user organizations move deeper into analytics (as a complement to reporting), they also move
deeper into real time. For example, this report’s survey ranks operational BI and reporting as leading
practices today, with 43% of respondents using these with some level of real time. However, 34% of
respondents are already practicing operational analytics in real time, with an additional 34%
planning to adopt the same within three years.
Other areas that should experience strong adoption within three years include real-time data
warehousing (and similar practices such as active and dynamic warehousing), predictive analytics,
visualization, social media analytics, text analytics, CEP, and stream mining.
Most of the data handled in real time today is structured (or more specifically relational), followed by
application logs and semi-structured data, but survey respondents anticipate more real-time handling
for social media data, Web logs and clickstreams, and unstructured data.
This report helps users understand the many available technologies and practices for real-time data,
BI, and analytics, plus how these enable fast-paced business processes and yield a wide range of
organizational advantages.
Actian, Cloudera, Inc., Datawatch Corporation, HP Vertica, IBM,
Tableau Software, Treasure Data, Inc., and WebAction, Inc.,sponsored the research and writing of this report.
About the Authors
Philip Russom, Ph.D., is senior director of TDWI Research for data management and is a well-known figure in data warehousing, integration, and quality, having published over 600 research reports, magazine articles, opinion columns, and speeches over a 20-year period. Before joining TDWI in 2005, Russom was an industry analyst covering data management at Forrester Research and Giga Information Group. He also ran his own business as an independent industry analyst and consultant, was a contributing editor with leading IT magazines, and a product manager at database vendors. His Ph.D. is from Yale. You can reach him by email ([email protected]), on Twitter (twitter.com/prussom), and on LinkedIn (linkedin.com/in/philiprussom).
David Stodder is senior director of TDWI Research for business intelligence. He focuses on providing research-based insights and best practices for organizations implementing BI, analytics, data discovery, data visualization, performance management, and related technologies and methods and has been a thought leader in the field for over two decades. Previously, he headed up his own independent firm and served as vice president and research director with Ventana Research. He was the founding chief editor of Intelligent Enterprise where he also served as editorial director for nine years. You can reach him by email ([email protected]), on Twitter (twitter.com/dbstodder), and on LinkedIn (linkedin.com/in/davidstodder).
Fern Halper, Ph.D., is vice president and senior director of TDWI Research for advanced analytics. She is well known in the analytics community, having been published hundreds of times on data mining and information technology over the past 20 years. Halper is also co-author of several Dummies books on cloud computing and big data. She focuses on advanced analytics, including predictive analytics, text and social media analysis, machine-learning, AI, cognitive computing, and big data analytics approaches. She has been a partner at industry analyst firm Hurwitz & Associates and a lead data analyst for Bell Labs. Her Ph.D. is from Texas A&M University. You can reach her by email ([email protected]), on Twitter (twitter.com/fhalper), and on LinkedIn (linkedin.com/in/fbhalper).