Are You Ready for Operational Intelligence?
A new TDWI report assesses operational intelligence in terms of its applications, its potential benefits, its people, process, and technology requirements, and its challenges.
- By Stephen Swoyer
- October 8, 2013
We've been hearing much more about operational intelligence (OI), a practice that analyzes logs, events, and messages at real-time or near-real-time speeds.
A new report from TDWI Research assesses OI in terms of its applications, its potential benefits, its people, process, and technology requirements, and -- of course -- its challenges.
As TDWI's Philip Russom notes, OI can be especially challenging from a data management (DM) perspective, chiefly because even the handiest tools in the DM toolset are ill-equipped to deal with its requirements. "One of the challenges is that traditional BI/DW tools were not designed for these new data sources and data types," writes Russom, research director for data management with TDWI, in Operational Intelligence: Real-Time Business Analytics from Big Data, the latest entry in TDWI Research's "Checklist Reports" series. "BI/DW tools are certainly not going away," Russom stresses, "but there's a need to complement them with new technologies for the new sources of big data, and operational intelligence supports this growing need."
On paper -- or, rather, in digitized 0s and 1s -- OI looks like an especially compelling technology. Vendors such as Splunk Inc., Vitria Inc., and others tout OI as a means to identify potential issues before they become problems, detect and monitor anomalies, and optimize delivery or distribution networks. OI also aims to help businesses identify and exploit time-critical opportunities. "The point of operational intelligence is to gain insight into new data sources so that business opportunities, organizational threats, and performance issues are detected and addressed as soon as possible, thereby enabling reactions that leverage or correct a given situation," Russom writes.
He cites several concrete use cases, from real-time fraud detection and mitigation to "smart grid" capacity management for utilities. In the latter case, a utility would use OI to identify or predict excess capacity in order to sell it to another utility company. Still another application involves using OI to monitor customer behaviors in real-time across the Web, social, and mobile channels.
Traditionally, organizations used a combination of business intelligence (BI), customer relationship management (CRM), and predictive analytic tools to (try to) support these applications. Traditional scenarios dealt primarily with in-house data, or purchased from third-party sources. In any case, with structured data, usually sourced from relational systems, OI proposes to make use of "multi-structured" data from a variety of sources, Russom notes.
In this respect, he sees it as a complement to BI and the data warehouse (DW). "BI/DW originated to support business decision making from structured data sources and to provide analytics from a historical perspective. OI complements BI/DW by providing insight into new unstructured and semi-structured data in real time. It handles big data in ways BI/DW cannot," he writes.
In many cases, OI systems also consume and analyze information from the data warehouse, Russom points out. "Data of different latencies tells managers different things about a business entity, such as a customer, transaction, or business process," he writes. "OI can correlate the real-time analysis of streaming big data and machine data with historical data -- typically managed in a data warehouse or similar database -- to present a complete view."
As Han Solo might point out, however, deploying, managing, and using OI -- much like flying through hyperspace -- isn't like dusting crops. OI requires real-time or near-real-time feeds to be useful. The events, messages, and logs that OI parses or analyzes typically aren't structured in a tabular format, which means that most OI offerings must use a streaming DBMS platform.
"OI's real-time analytics and business monitoring depend on correlations across many sources of big data that are inherently streaming, typically clickstreams from Web servers, machine data, data from devices, CSV, events, transactions, and customer interactions," Russom writes.
"High counts of small messages or events can add up to big data. A successful solution for OI must do several things with streaming data. OI must capture each event from a stream, separate events of interest from noise, make correlations with other streams and databases -- including data warehouses -- react to some events in real time, and store events for offline analytics."
Streaming platforms tend to be highly optimized: conventional RDBMSes, along with other conventional DBMS platforms, are insuffiicent, as is Hadoop, which is ill-suited as both a repository and as a platform for real-time analytics. In most cases, events, logs, and even messages can be reduced to tabular format for ingestion by a relational database, but this requires an ETL first-pass. In real-time or at near-real-time speeds, this just isn't practicable. Both RDBMSes and Hadoop are better suited for historical analysis of this data, which is another important function of OI.
In addition to handling eclectic types of streaming data in real time or near-real-time, an OI platform must also be able to scale -- big time. This is because the sizes of the data sets OI works on are much larger than are those of traditional decision support. "Operational intelligence ... handles data in extreme environments, where big data volumes are counted in terabytes and real-time data is generated in continuous streams," Russom writes, noting that big data is primarily "big" because it's compounded from a diversity of sources: "This includes new frontiers, such as sensor data and machine data, plus other frontiers such as unstructured data [e.g., human language text] and multi-structured data [e.g., XML, JSON, and CSV documents]. Traditional data types are still with us, too, in the form of structured data, relational data, and record-oriented flat files."
Russom's report also considers OI and machine data -- which he says should be persisted for historical analysis -- which he says "contributes to 360-degree views for a more accurate picture" of business operations. It also devotes time to a consideration of how streaming data can be used to complement or enrich structured, in-house data (and vice-versa).
You can download the report here. (A short registration is required for readers downloading a complementary TDWI report for the first time.)