TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

Data Integration Goes 3D

What is Wherescape 3D and why is it free? One industry watcher say it's unlike anything he's aware of in the market.

• By Stephen Swoyer
• October 4, 2011

WhereScape 3D seems to turn WhereScape Inc.'s traditional value proposition neatly on its head. The company positions 3D as a kind of test laboratory for data integration. It's described as in perpetual beta -- much like Gmail, which Google Inc. didn't promote to a "production" application until mid-2009.

Right now, 3D can be used to test -- and, at some point in the not-too-distant future, will be able to drive -- projects for Red, WhereScape's flagship data integration (DI) offering. The best news: 3D is free.

"Our idea behind 3D is, how soon can you get to a point where you can accurately fix-price the amount of work you're going to do?" says Michael Whitehead, WhereScape's founder and CEO.

3D is the product of WhereScape's dissatisfaction with the data warehousing (DW) status quo, according to Whitehead.

"People would say to us, 'How long is this [project] going to take?' and this was frustrating for us, because people wanted to give us work, and they wanted to budget for us appropriately, but the quickest answer was for them to build it [in RED]. The way a lot of [customers] get approval for budgeting, that couldn't be the quickest answer," Whitehead explains.

Ultimately, Whitehead says, the plan is to equip 3D to generate output for other environments, such as DI tools from Oracle Corp. or Microsoft Corp.: "We basically want to know: What do we want to have in the product that makes it a gold standard for planning a data warehouse planning project."

3D's test laboratory aspect isn't what's so dramatically different -- it's 3D's potential use as design or modeling software -- as, in effect, a planning tool for DI.

WhereScape, after all, is best known for RED, its quick-but-not-necessarily-neat ETL tool. It isn't that RED generates inelegant or dirty ETL code; WhereScape officials (and the many WhereScape customers the company could connect you with) would vehemently dispute this. It's rather that RED -- one of the first agile data integration (DI) tools -- is premised on a code, show, and go model in which data management (DM) teams, collaborating with business stakeholders, work to iteratively evolve a DW infrastructure. RED doesn't promise to get it right the first time; it promises to get it right over time -- or, claim WhereScape officials, more quickly and/or flexibly than traditional DI tools.

It works like this: a DM team uses RED to quickly generate ETL code for a use-case or requirement, shows the result to business stakeholders, tweaks RED based on their feedback, and -- always and iteratively -- evolves the warehouse.

With 3D, WhereScape is telling its customers that they might could put a little bit more planning into this process.

The concept of planning isn't altogether anathema to that of agile software development or agile BI, however. Customers could use 3D in place of the proverbial white board that's commonplace in (for example) so many agile software development projects. 3D does have the ability to capture whiteboard-like jottings, diagrams, or other brainstorming assets as part of its modeling and planning facility.

However, 3D isn't just a planning tool. To the extent that it can be used as a test or modeling laboratory for data integration, in fact, its most intriguing use-case could be feasibility testing: e.g., is such a project possible, and -- more important -- at what time and at what cost?

That's how Ahmad Malik, director of IT application services for Learning Care Group (LCG) Inc., plans to use it.

Malik, an existing RED customer, says he hopes to use 3D to assess the feasibility of a proposed project involving LCG's CRM systems.

Absent 3D, Malik says he wouldn't have any way to benchmark the costs -- to say nothing of the potential ROI -- of this project. "I know I'm going to change the CRM system, I'm going to change [the system] in two steps. One is initially to make changes in the system [itself] to get all of the data to detail level. The second is going to happen over two years as money frees up, and that's to replace the system. I want to see how 3D can help me plan for this. Hopefully, it can even help make the [business] case to accelerate [the process]," said Malik, interviewed at TDWI World Conference in San Diego last month.

Malik spoke at TDWI's Executive Summit, primarily discussing LCG's use of WhereScape RED to quickly deploy its new data warehouse environment.

He hasn't yet used 3D and probably won't actually get a chance to put it through its paces until the end of August. There's little to no risk, as Malik sees it: he's already using RED, 3D is free, so why not see what it can do?

"It will be interesting to see if [3D] can help us estimate that project," he comments. "Our initial estimate is about three months, but part of the reason we think it will be high is that there's a bunch of unknowns. We're hoping that [3D] can help us answer a lot of those unknowns."

Mark Madsen, a principal with BI consultancy Third Nature Inc. and himself a veteran data warehouse architect, says that 3D is, for all intents and purposes, one of a kind.

"The tools that I've seen out there are the front-end design tools that the guys like BusinessObjects [used to] try to bolt on [to their DI tools]. It was the kind of design interface that would let sort of a business analyst spec out something and see, you know, 'it comes from there and it goes to there.'"

Other vendors copied (or copied and improved upon) these approaches, Madsen says, but no vendor has yet developed anything quite like 3D.

Quite aside from 3D's use as a planning or testing tool, Madsen also believes it could be valuable as a BI development tool.

"Generating data is a really useful thing for development in general. To this point, I think that only [Microsoft's SQL Server Integration Services] can generate for you data along particular tests -- you know, just to generate really simple tests, like, if you wanted to test 500,000 rows of transaction data," he observes. "I can see it being useful for both testing and for BI development."