Why Specialty Data Warehouse Systems Are Here to Stay
TDWI survey and analysts agree: analytic databases have arrived -- officially.
- By Stephen Swoyer
- March 31, 2010
Netezza Inc., Greenplum Software Inc., Kognitio, and other analytic database players helped revitalize an otherwise sleepy data warehouse (DW) segment, but they've also generated a lot of questions: To use an appliance or not? To use a columnar database or stay with the status quo?
Other key questions arise as well: Are specialty analytic databases destined to replace conventional DBMSes? Are analytic database systems more hype than substance? Is there even a bona fide demand for next-generation data warehousing platforms?
Almost a decade after Netezza’s founding -- and nearly half a decade after competitors DATAllegro Corp., Greenplum Software Inc., and Kognitio entered the specialty DW segment -- analytic databases are here to stay.
Consider a recent report from TDWI -- Next Generation Data Warehouse Platforms -- which found that adopters increasingly view analytic database platforms as strategic bets.
In a February report, IDC researcher Carl Olofson went even further, projecting that by 2015, column-based databases will predominate in data warehousing scenarios, and that in-memory architectures will likewise augment (or perhaps even supplant) conventional DBMS systems over the same period.
The salient takeaway, according to Olofson’s analysis, is that the new (or “third”) generation of database platforms is even now transforming the market.
These are heady times for specialty database players. The typical analytic database deployment scheme -- one once championed by Netezza, Dataupia Inc., and ParAccel Inc. (among others) -- used to involve tactical deployments: e.g., to address persistent application performance issues; as (dependent or independent) data marts; and so on. Not anymore.
According to TDWI’s survey data, for example, more than half of all shops (51 percent) expect to augment or replace their existing data warehouses with a specialty DW system at some point over the next five years.
That jibes with IDC’s projection, although TDWI stops short of endorsing any particular architecture -- columnar or row-based -- as an analytic heir-apparent.
Change DW Pros Can Believe In?
According to TDWI’s survey, fully one-fifth of respondents expect to augment or pull the plug on their conventional DW systems this year.
The report’s author -- Philip Russom, senior manager at TDWI Research -- cites a number of concerns he says are helping drive demand for analytic database technologies. For starters, there’s surging demand for real-time or right-time connectivity -- and, specifically, for a new class of on-the-fly analytic technologies designed to deliver results in real- or right-time.
Russom points to spiking data warehouse volumes that -- according to more than one-third (37 percent) of survey respondents -- are stressing conventional DW systems to their breaking points. Other technologies having an impact: the mainstreaming of 64-bit architectures, which introduce an effectively limitless (256 TB, in the case of current-generation Xeon silicon) memory address space and an emerging preference for massively parallel processing (MPP) DW architectures.
By far the biggest driver, according to Russom, is demand for improved analytic horsepower, which is itself a product of growing popularity of advanced analytic technologies. The simple fact of the matter, Russom says, is that many conventional data warehouse implementations were designed chiefly to address reporting or basic online analytic processing (OLAP) requirements.
The rub, experts point out, is that such DW implementations -- which are often based on shrink-wrapped DBMSes -- don’t have the innate analytic horsepower to support advanced analytic requirements. Olofson, for example, cites the analytic shortcomings of conventional DBMSes as the primary impetus for the growing popularity of (and projected market preference for) column-based databases. Similarly, Russom says, fully 40 percent of respondents in TDWI’s survey cited the shortcomings of their existing DBMS platforms, particularly with respect to analytic requirements, as an impetus for analytic database adoption.
Russom stresses that a range of issues also factor into deployment decisions. “There are multiple forms of advanced analytics, including those based on data mining or statistics and those based on complex ad hoc SQL statements. The former may or may not run in a DBMS -- depending on the vendor’s analysis tool capabilities -- which is a problem when it forces users to move data out of the data warehouse for the sake of analysis, then back in,” Russom explains.