Agile Meets Integration at TDWI Conference
At last week's TDWI Summer World Conference in San Diego, bottom-up disruption -- of an agile kind -- was a salient theme.
- By Stephen Swoyer
- August 25, 2010
At last week's TDWI Summer World Conference in San Diego, bottom-up disruption -- of an agile kind -- was a salient theme.
Agile business intelligence (BI) doesn't have to have a bottom-up thrust, of course, but for many vendors at last week's conference, agility definitely starts in the trenches.
Take BI integration firm Kapow Technologies, which champions a pragmatic take on data integration: if data can be exposed in the context of a Web browser, Kapow promises to get to it. The company says their technology can be used as a quick-and-not-entirely-dirty means to integrate publicly-available or subscription data and or as a means to integrate data from internal sources (including portals or Web-enabled legacy applications) that IT hasn't yet brought into the data warehouse.
From the standpoint of traditional data management (DM) practitioners, what Kapow does and how it does it is important. Ron Yu, Kapow's vice president of marketing acknowledges that Kapow's quick-and-not-quite-dirty approach to integration (which is anchored by its Kapow Web Data Server product) can sometimes be a tough sell, at least to DM purists. That being said, Yu urges, a tool such as Kapow helps DM become more responsive and more nimble -- more agile, if you will.
He cites the experience of German auto giant Audi, which implemented Kapow to accelerate a stagnant data integration effort. Audi's DM team had hit a series of brick walls in an effort to negotiate access to data hosted by several partners (prominent German and Italian automotive manufacturers): the German and Italian DM teams put up resistance, for one thing, while -- further down the line -- various people and process issues likewise intervened.
"In a [data] integration project like this, the technology [integration] can sometimes be the least of your worries. You have resistance from all of these stakeholders who tend to jealously guard the data or resources that [they feel] they own," Yu explains. "It can bring everything to a halt. Meanwhile, the business users are going, 'We just want to be able to [run analytics against] this data that we already have access to [via a Web browser] or that we're already paying for.'
"Kapow cuts out the middle man. If it's in a Web browser, if it can be displayed in a Web browser, we can get to it. We can get to it cleanly [and] elegantly -- and we don't have to worry about access permissions or firewalls or security."
Ditto for extract, transform, and load (ETL) specialist WhereScape Inc., which launched a new release of its flagship WhereScape Red ETL tool at the conference. Longtime ETL powers such as IBM Corp. and Informatica Corp. like to tout their data integration (or, increasingly, enterprise information management) bona fides, but WhereScape insists on its ETL-ness. More to the point, says CEO Michael Whitehead, WhereScape delivers a kind of tactical agility.
"It's a dirty little secret of the industry. For too long we as an industry have been ignoring the fact that we take too long to build things, and this is because we assume [at the beginning] that we have to get them right -- but we don't ever get them right! That's why you see people still doing satellite data marts, still doing their own Excel spreadsheets, still doing [operational data stores]: it's because IT can't get it right the first time," he contends.
Whitehead's argument isn't heresy. In a presentation at TDWI's Las Vegas Executive Summit, for example, data warehousing expert Mark Madsen singled out WhereScape's Red (along with similar offerings from competitors Expressor Software Corp. and Illuminate) as alternatives to conventional DI solutions. More to the point, said Madsen, tools such as Red eschew the static or inflexible data models that tend to constrain conventional DW implementations. Instead of exhaustively scoping and defining a data model upfront, you could use a tool such as Red to effectively generate your DW.
If or when your requirements change, you'd regenerate -- or, in the language of agile software developers, refactor -- your warehouse. "That's the good thing [that] the agile community brings to the data warehousing world -- the positive side of it is that they're coming from the assumption that you can't understand everything upfront: you can't get it all right in the beginning, so you [instead] do it as a small series of deliverables," Whitehead comments.
Elsewhere on the integration front, data federation players Composite Software Inc. and Denodo Technologies Inc. likewise stepped up to seize the agile mantle. This isn't a stretch, either. When federation tools first appeared, after all, they were sometimes touted as killer apps for data warehouse prototyping.
Similarly, federation was both touted as a killer app for frequently refreshed reporting and derided (chiefly by data management purists) as a stopgap or Band-Aid solution for "virtual" data warehousing.
These days, says Bob Eve, executive vice president of marketing with Composite, the DM Holy Wars are over. Most of the hype has died down, and federation -- which once aspired to an acronym of its own (Enterprise Information Integration, or EII) -- is an accepted and largely non-controversial DM technology. Although it's still used for DW prototyping, for frequently refreshed reporting, and (occasionally) as a stopgap "virtual" data warehouse, its fastest growing use case is as an enabling technology for data virtualization.
"Organizations are dealing with data of extreme complexity and with data volumes that continue to explode. They're also trying to quickly incorporate new technologies -- especially predictive analytics -- and service the increasingly sophisticated needs of users. More important, there's this sense that while the [enterprise data warehouse] is nice, it isn't necessary. I don't know that you'll find anyone seriously arguing that the EDW is essential anymore," Eve observes.
In this respect, data federation software comprises a pragmatic solution. "The concept of the virtualized data layer, which says instead of bringing everything into one place [as with an EDW], you can just leave it where it is but still make it available to your users."