When Right Time Isn’t Real Time Enough
The bottom line: companies need to figure out a "right time" window that’s right for them.
- By Stephen Swoyer
- May 30, 2007
When is right-time not real-time enough? That, as it happens, is the million-dollar question. Business intelligence (BI) and performance management (PM) vendors have made an awful lot of noise about a "new generation" of "real-time", "near-real-time", or "decisioning" tools, touting everything from their ability to provide "actionable business insight" to other intangible (and usually unspecified) competitive advantages. Industry watchers, on the other hand, like to stress that there is no single "real-time" Rx for every company or every industry; instead, they argue, companies need to figure out a "right time" window that’s, er, right for them, and go from there. Who’s right? Who’s wrong? Does it matter?
Well, everyone. And no one. And, yes, it does matter. In fact, once you get down to brass tacks with most BI and PM vendors, you’ll find that their views are actually very similar—for the most part identical—to those of many prominent industry talking heads.
Philip Russom, director of research with TDWI, succinctly frames the issue. "Call it what you will: real-time, near-time, right-time, on-demand, operational BI, inline analytics, embedded reporting, real-time data warehousing, active data warehousing or dynamic data warehousing. It’s all about satisfying one bone-crushing business requirement: When time-sensitive information originates or changes in an operational system, how do you synch that with a data warehouse and then push out the information—complete with a performance history, as kept by the warehouse—to business end-users who need to know ASAP?" In any of its flavors, Russom argues, "real-time" has proven to be a surprisingly tricky beast to tame.
"Moore’s law and cheap storage cured the scalability crisis. The cure for high-speed information delivery is far more complicated, because it requires that a long list of integration and BI components—many newly enabled for services—be added to the already complex data warehouse technology stack, and sometimes operational technology stacks, too."
Amy Meyer, director of data quality product marketing with Business Objects, outlines a real-time/right-time problem set that isn’t much different from Russom’s. "Many customers say they want real-time [information access], but they’re also intimidated by some of the technical challenges. In many cases, though, [real-time] really isn’t what they need," Meyer said in an interview at February’s TDWI World conference.
What they need, she says, is typically a shorter time-to-refresh window. And for many customers, Meyer points out, "right-time" is going from a once-a-week or a bi-weekly refresh cycle down to a nightly one. In other cases, she indicates, perhaps a near-real time approach (such as trickle-feeding data into a warehouse) might be both technically feasible and operationally desirable.
The real-time/right-time challenge is spurring BI and PM players to explore non-traditional solutions, too. Meyer points to Business Objects partnership with data warehouse appliance specialist Netezza Inc. to promote a turnkey rapid information access solution for SAP customers.
"SAP customers are looking for a fast way to get access to the information in their SAP ERP applications so they can do analysis on that, they can do reporting, they can track metrics," she comments. "So what we have jointly developed with Netezza is an offering that prepackages a solution to give them fast access. It includes [data integration], which we’re bundling with four new Rapid Marts."
Rapid Marts are a Business Objects staple, based on an ETL feature it acquired from the former Acta. For the record, Acta specialized in SAP connectivity. That lineage gives Business Objects a direct connection into the very innards of the SAP ERP stack and enables both a rapid time-to-deployment and a rapid information access paradigm, too.
"We understand the data in an SAP environment, how to pull that information out quickly, what transforms need to be done on that data—we know how to do that," she commented. "And with Netezza, we are specifically targeting SAP as the source for sales, accounts receivable, accounts payable. Netezza being the data mart, what we can offer is not only the performance you get with an appliance, but also we’re able to get them up and running in weeks rather than months."
Michael Corcoran, vice-president of product marketing with Information Builders Inc. (IBI), says what’s "right-time" for a particular organization really comes down to a trade-off between both complexity and expense and the value that the business places on frequently refreshed data. Or, to put it another way: if you have to ask how much it costs, real-time probably isn’t right-time for you.
"It really comes down to the customer’s interpretation of what is or what isn’t ‘right time.’ From a customer who just went from a monthly down to a weekly or even a nightly feed, they’re feeling pretty good about that," Corcoran explains. "Most people are still doing most of their BI off of 24-hour data warehouses, although increasingly they’re feeding it maybe in chunks throughout the day so it’s near-real time." On the other hand, he continues, real-time probably isn’t as expensive a proposition as many customers think.
"I think what a lot of organizations don’t realize is that if you look at the real-time integration technologies, you really could trickle-feed it on a real-time basis; as I’m recording this acquisition, I really could trickle-feed it to the data warehouse. That’s really doable now," he argues.
Corcoran and IBI have a dog in the real-time/right-time stakes, of course: Information Builders’ iWay Software subsidiary develops and markets a wide variety of integration adapters. Corcoran concedes as much, but says that his broader point still stands: the integration game has come a long way since the days of largely batch-oriented ETL (or scripted FTP data loading).
In addition to ETL, there’s enterprise information integration (EII), the service-oriented architecture (SOA), Web services pervasiveness, and other improvements, too. Add it all up, Corcoran argues, and you’ve got a mise-en-scene in which real-time or near-real-time information access has become increasingly right-time for many organizations.
Corcoran continues: "Really that whole wall between the worlds of BI and application integration is becoming the same. Traditionally, when people said, BI, it means, ‘I need an ETL tool and I’ve got to move all of my operational data into a data mart or a data warehouse.’ We have an adapter system that says, even if it’s in a legacy system or it’s not organized for reporting, we can give you an adapter so that you can report directly off of it," he comments. "You have this EII capability where you can join operational systems together. You have that EII capability, you’ve got this process-driven mechanism, you’ve got SOA, which from a BI perspective means that I can go across the internet and pull in real-time information as needed."
There’s a further wrinkle here, too. Thanks to the attentions of IBM Corp., Microsoft Corp., Oracle Corp., and Sybase Corp., data integration—or its ETL component, anyway—has become increasingly commoditized.
One upshot of this has been a kind of technology trickle-down: features or functions that were once part-and-parcel of high-end ETL suites are now embedded in RDBMS software. Another aspect is that many discrete BI and PM solutions now feature canned connectivity—based on ETL, EII, or plain old ODBC/JDBC—into relational and other data sources.
Consider the case of Relational Solutions Inc. (RSI), a services and consulting firm that markets its own data integration and—more recently—PM tools. RSI has a dog of sorts in the real-time/right-time race, too: its new BlueSky Performance Books product. Performance Books boasts a slick, Flash-powered user interface: it organizes KPIs by tabs; displays its graphs, charts, and other indicators in a virtual notebook view (complete with a spiral ring at its center); lets users dynamically drag, shuffle, or rearrange performance indicators; supports interactive drill-down; and integrates with Microsoft Excel. Big deal, right?
After all, Flash-powered PM solutions are a dime-a-dozen, these days—or so it seems, anyway. Not according to RSI co-founder and COO Janet Dorenkott. She cites a roster of competitive differentiators—including built-in ETL connectivity—which help set Performance Books apart from the PM main. "Performance Books can be driven from a data warehouse or a data mart, or any other data source, so I can set this up to go after an ERP system if I wanted to," Dorenkott indicates. "It’s very cost-effective. You don’t need to have a massive infrastructure in place to support this, because there’s a data integration piece built right in."
Data integration comes courtesy of a canned version of BlueSky Integration Studio (BIS), RSI’s own ETL tool, which boasts native connectivity into both Oracle and SQL Server, as well as connectivity (via ODBC or similar technologies) into other data sources, too. BIS ships with a drag-and-drop ETL studio—complete with support for EDI, AS2, and other industry-specific (or esoteric) standards—which isn’t entirely replicated in Performance Books.
On the real-time tip, RSI is also developing a trickle ETL capability for Performance Books, too. This will give decision-makers near-real-time access to performance information, says RSI co-founder and president Rob York. "With this [trickle ETL feature], you can create a performance book chart and it’ll update every 15 to 20 minutes. You can literally specify how long or short you want the [update] window to be when you’re creating it [the ETL job]," York says.
Trickle ETL isn’t for everyone, York acknowledges, and it has its limitations—at least in terms of the size and scope of trickle data—but it’s of increasing interest to many customers. "They’re going to want to make sure that it’s a very targeted data set and it’s very small. They can’t trickle everything [in the warehouse]. But if you’re interested in [doing trickle ETL], you obviously see a need to give [decision-makers] data that’s changing up to the minute or up to the second. Previously, something like this was just too complex or too expensive. But as long as you’re [trickling] a very targeted data set, it’s certainly feasible."