Big Data Challenging the Status Quo
Big data is about mixing, mashing, and commingling all kinds of (seemingly unrelated) information together. This has big implications for BI and data warehousing.
- By Stephen Swoyer
- November 27, 2012
The exhibit hall at this year's Strata + Hadoop World conference looked like a surrealistic tableau, with its juxtaposition of seemingly incongruous objects -- or, in this case, vendors.
Strata + Hadoop World gave us old-school vendors such as Versant Corp. -- the venerable object database company -- along with wunderkind upstarts such as SiSense; established players such as Simba Technologies Inc. -- the ODBC and JDBC connectivity specialist -- shared floor space with machine data powerhouse Splunk Inc.
Even Microsoft Corp. was there.
Big data is, in a way, rebooting the status quo -- time-tested assumptions are being tested, evaluated, and (in some cases) discarded. In the data warehousing (DW) world, this means that the arrangements or practices that worked so well for so long -- or which seemed so self-evident or common-sense -- are now being called into question. These practices might not themselves disappear (or become extinct), but at the very least they'll be reevaluated -- especially if they're products of the SQL-centrism of the DW world.
For a company such as Versant, which has no business intelligence (BI) or DW presence of which to speak, this means highlighting the way object support is implemented in most SQL-centric relational database platforms.
"The analogy I've used in the past is if you have what is predominantly a relational model and then you have certain types of data that don't fit into that, what does a RDBMS do? Instead of putting your car in the garage, you put a note in [the garage] saying that your car is parked outside," says Dr. Robert Brammer, Ph.D., independent director of Versant.
Brammer is referring to BLOBs, or binary large objects, the standard way of accommodating unstructured data -- such as photo or video objects -- in an RDBMS.
"You have to put some kind of pointer out with some kind of kludge as to how you're going to handle this. Some of these national intelligence systems, they're all BLOBs," he says.
Brammer knows about national security. Before joining Versant, he was CTO of Northrop Grumman Information Systems. He's also a principal with Brammer Technology, a Boston-based information management consultancy.
Neither Brammer nor Paul McCullugh, Versant's executive vice president of sales, wants to champion an object-oriented data warehouse. Brammer, in fact, rolls his eyes at the notion.
At the same time, he argues, BLOBs are kludgey. They're kludgey because -- in the relational era -- kludgey was acceptable, at least for unstructured information: from the perspective of SQL-centric data management (DM) teams, unstructured information was viewed as incidental, as peripheral; as a second-class citizen in comparison with the elegant structure of relational information. Think of it as a kind of information apartheid, in which structured SQL was accorded certain privileges -- vested in the SQL-native data warehouse -- that were denied to unstructured data. What's most intriguing about 'big data,' says Brammer, is that, as an industrywide phenomenon, it has the potential to explode information apartheid.
It has the potential to break the power, the exclusivity, of the data warehouse.
Take Hadoop. It's polyglot: in addition to SQL, Hadoop speaks (or can be made to speak) Perl, Python, Java, C++, and a host of other languages. Unlike the data warehouse, which wants to speak only SQL, Hadoop is inclusive: its motto could easily be "there's a library or a project or a framework for that." In a series of interviews at Strata + Hadoop World, several attendees contrasted Hadoop's inclusivity -- its essential pragmatism -- with what they described as the exclusivity (the essential recalcitrance) of the data management world.
A representative with a prominent data integration vendor told BI This Week about a customer that wants to use Hadoop to bypass its DM team altogether.
"It's almost like a coup d'etat for them. They asked [their application developers] how long it would take to develop this [specific] application. Eight days, they were told. That's great, but when they asked how long it would take to get [data] source connectivity for these [applications]? Nine months. They were told nine months. Now they just want to go around [the data management group]," this representative said. "Basically, what they want is an ETL layer for all source data and they basically want Hadoop to be their new massive data warehouse."
As the teeming exhibit hall at Strata + Hadoop World demonstrated, big data is about mixing, mashing, and commingling all kinds of (seemingly unrelated) information together in a single big bowl: on the show floor, for example, some of the best-known names in the BI and DW space -- Greenplum (a division of EMC Corp.), IBM Corp., Kognitio, Oracle Corp., ParAccel Inc., Pervasive Software Inc., SAP AG, SAS Institute Inc., Talend Inc., and Teradata Inc. -- mixing and matching with open source software (OSS) services vendors, application development tools vendors, hardware and hosting vendors, and others.
One upshot of this is that many age-old distinctions -- e.g., the explicit and often antagonistic separation between application development and data management -- are starting to become less important, less definitive, and less exclusive. The data warehouse will still have its place, just not the place. It will be one (important) source among many. First among equals, so to speak.
"I think a lot of what is becoming more widely appreciated now is that the earlier data models are being stretched past the point of breaking. This is encouraging us to look more closely [at these earlier models]. You used to have a relational model, perhaps with some sort of object-relational data mappings [i.e., BLOBs], and this was acceptable. It wasn't ideal; it was acceptable. It isn't acceptable anymore." says Brammer.
What's driving this, he continues, isn't so much the problem of big data -- i.e., the so-called 3 Vs -- but the opportunities presented big data-powered products, services, and processes.
"If you look at the new generation of 'Smart Grid' problems, there are people who want to put a lot more advanced information technology into the power grid to make it much more efficient and environmentally-friendly and more cost-effective, but that means they're going to put a lot more instrumentation out there that's going to collect a lot more data, much of which they're not currently dealing with at all today. This is very likely a growth area for Versant, as a high-performance [object-oriented] engine for this [kind of use-case]."