TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

As Right Time Approaches Real Time Data Integration Is Key

At bottom, real-time is a data integration problem: it involves getting fresh data to business users as rapidly as possible.

• By Stephen Swoyer
• May 30, 2007

When you get right down to it, the real-time/right-time challenge is a data integration problem. It involves getting fresh, actionable data to business users (typically at all levels of an organization) as rapidly as possible.

Not surprisingly, the big data integration players are at the forefront of the real-time/right-time push. One upshot of this is that the data integration powers-that-be have evolved in striking ways over the last half-decade. Or, to put it another way: data integration doesn’t begin and end with ETL anymore.

Consider Informatica Corp., largest (or most prominent) of the standalone data integration vendors. Informatica’s special sauce has always been ETL, but—over the last 36 months, at least—the company has buttressed its solution stack with a number of complementary technologies, including data quality (courtesy of its acquisition of the former Similarity Systems), enterprise information integration (EII)—thanks to a long-standing agreement with Composite Software—software-as-a-service, and unstructured data access and management (via its acquisition earlier this year of ItemField).

Elsewhere on the information integration tip, Informatica has relationships with prominent enterprise application integration (EAI) providers, too. The upshot, officials argue, is that Informatica can prescribe an appropriate Rx for just about any information integration ailment.

And that’s regardless of a company’s size, too, says Informatica director of product marketing Don Tirsell. He cites his company’s growing SaaS stack, and says SaaS simultaneously lets Informatica reach out to small- and medium-sized enterprise customers, but can help accelerate information delivery for large customers, too.

“[SMB customers] face a lack of infrastructure and resources to tackle some of the big integration problems,” he says. “They tend to focus on the silo-ed application approach, because they have specific business needs. So as integration issues arise, they don’t have the resources or they aren’t sophisticated enough in some cases to deal with them. A hosted offering is attractive to that market, because they don’t have to set everything up. We’ve taken the breadth of our tools out of the way and made it a very straightforward wizard-driven process [where there’s] little training and it’s cost-effective.”

On top of this, Tirsell argues, SaaS’ rapid time-to-deployment, and—for customers who opt for complementary SaaS offerings, such as Salesforce.com—rapid information access comprise strong selling points, too.

The salient point, Tirsell concludes, is that information access paradigms are a-changing, and that SaaS, EII, and other technologies should (or must) factor into any savvy customer’s calculus. “I think the industry is maturing to the point where there are known best practices, methodologies in a software platform that can handle the breadth of data integration projects,” he concludes. “These [practices] are on top of ETL, [and] they include hosted [SaaS] and EII [capabilities], and also data quality, and unstructured [data management], too.”

Furthermore, if the last few years have demonstrated anything, it’s that consolidation is alive and well in the BI and DW segments. Informatica itself pulled off two signature consolidations over the last twelve months (those of Similarity Systems and Itemfield). More tellingly, close Informatica partner Firstlogic Corp.—whose data quality technology was deeply integrated with Informatica PowerCenter—was acquired by rival Business Objects SA. With this precedent in mind, isn’t Tirsell concerned that onging consolidation could endanger some of Informatica’s existing relationships—including its crucial partnership with EII power Composite?

Tirsell is surprisingly nonchalant about this scenario. “We actually expect that to happen [Composite being acquired by another vendor], but we’re leveraging their product as it sits today and taking what they have and building around that. Our thinking is, let’s learn what they have and make it better and part of our own,” he comments. “We actually purchased Composite’s source code and we’re leveraging that as part of that core platform offering, but we’re also surrounding that [Composite source] with enhancements of our own. We can even pull in blocks of architecture in their platform. They have very sophisticated caching technology, which is very important for limiting the impact on operational systems, and they also have very sophisticated query optimization capabilities.”

Ditto for IBM, which became a full-fledged data integration player two years ago, with the acquisition of the former Ascential Software Corp. Since then, Big Blue has steadily worked to assimilate and extend its Ascential assets, rebranding them as WebSphere integration pieces. On the real-time tip, IBM earlier this year announced its most ambitious take on data warehousing to date, so-called “Dynamic Data Warehousing,” a technology vision that—surprise!—emphasizes real-time business decision-making. Big Blue contrasts its Dynamic warehousing vision with that of the largely batch-driven status quo, which it says injects an unacceptable amount of latency into the decision-making process.

“We believe that the world of traditional warehouseing is dead, really; the world of just looking in the rearview mirror,” said Karen Parrish, vice-president of BI solutions with IBM, in an April interview with TDWI Radio News. “The days have gone by where we looked at yesterday’s sales and compared them to same sales of last year and the year prior—don’t get me wrong, we still have to do that, but the world is an ever-changing set of transactions that cause us not only to have to look at what happened yesterday, but to have to determine based on what we know about what happened yesterday and the year prior, what exactly is going on today.”

Real-time isn’t just about giving customers lighting-quick access to actionable data, Parrish says. In fact, one important aspect of making data “actionable” is to present it in a context that’s appropriate for the user who is consuming it. As a result, she argues, an all-important piece of the real-time/right-time puzzle is not only giving users access to frequently refreshed data, but coupling that data with context-appropriate—i.e., intelligible—user-interface views.

“The real value of business intelligence is going to be when the average business user like you and I has the ability to see the information that we choose to see in the context in which we need to see it so that we can make better business decisions,” Parrish argued. “There’s nothing more aggravating than looking at sales figures or pipeline figures … that [are] not contextually relevant to you—[where] it’s somebody else’s view of how you should look at the data.”

Like IBM, other data integration powers-that-be have come into their own over the last few years. Consider Oracle Corp., which—not satisfied with fielding a best-of-breed ETL tool of its own (Oracle Warehouse Builder)—acquired ELT (or extraction loading and transformation) specialist Sunopsis Inc. last fall. A key driver for Oracle’s move? You guessed it: Sunopsis’ ability to offer real-time or near-real-time access to application data.

“This is in response to customers’ requirements for real-time access to information and support for heterogeneous environments. Sunopsis has broad support for a range of data products, including non-Oracle [platforms]. So we plan to focus on integrating the Sunopsis products with our products in [the Fusion Middleware] area—Oracle BI Suite, Oracle Data Hub, and our SOA suite,” said Rick Schultz, VP of Fusion Middleware with Oracle, at the time. “[Sunopsis] … reinforce[s] the Oracle Fusion middleware importance of providing hot-pluggable [access].”

A host of other data integration players, including SAS Institute Inc., Business Objects SA, Information Builders Inc. (IBI), and even Cognos Inc. (which acquired real-time dashboard specialist Celequest Corp. earlier this year) are touting real-time, near-real-time, or “right-time” data access capabilities. Could it be that real-time—like ETL itself—is on a fast-track to commoditization?

Probably not, industry watchers say: real-time describes a deceptively complex problem set, with many overlapping challenges, such that there probably isn’t a single silver bullet for any one company or any industry.