The Year in Review, The Trends Ahead
It's that time of year again. Every December, we look back at the year just past, highlight some of its most intriguing trends, and hazard a few guesses about what the New Year will have in store.
- By Stephen Swoyer
- December 15, 2010
It's that time of year again. Every December, we look back at the year just past, highlight some of its most intriguing trends, and hazard a few guesses about what the New Year will have in store.
So grab some eggnog, turn down (or altogether mute) your iPod's holiday playlist, put aside your iPad (unless, of course, you're using it to read this article), and join us as we take a look at the year that was 2010 -- as well as the year that could well be 2011. Hint: it's full of iPods and iPads.
As the Specialty Data Warehouse World Turns
At last … at long last. After more than a few false starts -- starting with Microsoft Corp.'s acquisition of the former DATAllegro in July of 2008 -- the data warehousing (DW) space experienced something very like consolidation this year. First, in June, SAP AG picked up Sybase Inc., proprietor of -- among other assets -- the suddenly resurgent Sybase IQ columnar database. Then, in July, storage giant EMC Corp. picked up veteran analytic database specialist Greenplum Software Inc.
Finally, in September, IBM Corp. nabbed the granddaddy of them all -- data warehouse appliance specialist Netezza Inc. -- for a cool $1.7 billion.
Specialty data warehouse players started getting gobbled up because big name players started paying attention -- or a lot more attention -- to Big Data, but in and of itself Big Data wasn't enough to move hardware. Netezza (and its competitors DATAllegro and Dataupia Inc.) had tried that, with mixed results.
It wasn't until Big-Data-the-Technology-Prescription got hitched to Advanced-Analytics-the-Killer-Application that the specialty data warehouse express really got going. In 2010, nearly everybody was talking about Big Data, usually in connection with some kind of advanced analytic application.
In a growing number of cases, that application was MapReduce. Aster Data Systems Inc. and the former Greenplum were first out of the gate with support for MapReduce (way back in September of 2008), but 2010 saw a rush of MapReduce-oriented activity, chiefly involving Hadoop, an open source implementation of the MapReduce framework. IBM, Teradata, Netezza, Talend, and Informatica Corp., among others, announced Hadoop integration plans.
The argument goes like this: The traditional data warehouse -- powered, in most cases, by a commercial, off-the-shelf (COTS) database package -- just isn't up to the task of managing and effectively crunching Big Data. What's needed is either a row-based data store powered by a massively parallel processing (MPP) engine, or -- even better, according to some -- an MPP-based columnar store. Although an MPP or columnar architecture lets you scale to manage Big Data, a technology such as MapReduce gives you the analytic horsepower to crunch Big Data. That's the version of the story that many vendors -- depending on their architectures, market niches, and (of course) differentiators -- like to tell.
A Dream Deferred -- Again
To most of us, the benefits of BI are so obvious that the enthusiastic adoption of BI technologies by users at every level in an organization seems like a no-brainer -- like something that must and will come to pass. Eventually. Inevitably.
"Inevitably" certainly is taking a long time, however. If anything, and in spite of the best-laid go-to-market efforts of the BI suite players, adoption has held steady.
In our end-of-year wrap-up last year, for example, we saw that BI adoption actually regressed. This year, several interests sought to draw new attention to this trend.
First, there was the Business Application Research Council (BARC), that continues to publish the BI Survey that OLAP market oracle Nigel Pendse first made famous. In the ninth edition of BI Survey, BARC found that only about 10 percent of enterprises have managed to deploy BI to a majority of their users.
In an earlier study, industry luminary Howard Dresner -- who's credited with coining the term "BI" -- highlighted another intriguing trend: the influence of IT in the BI buying process seems to be waning, according to Dresner, even as that of the line of business is increasing. Line-of-business customers are increasingly likely to adopt and (in some cases) even deploy BI tools without first soliciting IT's approval, according to Dresner.
This trend has encouraged ISVs to rethink the ways in which they develop, license, and market their software. It's likewise encouraged a kind of bottom-up counterrevolution -- let's call it an insurgency -- in BI and DW.
Insurgent BI: Anything Goes but the Status Quo
When all is said and done, 2010 might well go down as the year in which frustrated users started sharpening their pitchforks.
As a prescription to lagging user adoption, we've noted in previous articles a return to "workgroup" -- or end-user-oriented -- BI offerings. In one of the most intriguing trends of 2008 and 2009, several such offerings -- including LyzaSoft Inc.'s Lyza analytic workbench, QlikView from QlikTech Inc., Microsoft's then-gestational "Project Gemini" (the column-store-on-a-desktop add-in that Redmond designed for Excel 2010) -- either came to the fore (in the case of Lyza) or began making a good bit of noise (in the cases of both QlikView and Microsoft).
This year we saw what might be called an "insurgent" trend in BI, where upstarts such as Bis2 Inc., WhereScape Inc., Kapow Inc., and even Expressor Software Corp. (which effectively relaunched its flagship Expressor product in October) joined workgroup BI advocates in championing an anything-goes-but-the-status-quo approach to business intelligence, data warehousing, and even data visualization.
Veteran data warehouse architect Mark Madsen offered a neat summary of insurgent BI in his keynote address at February's TDWI World conference in Las Vegas.
Using the metaphor of a road-building project that paves over an existing -- albeit twisting and turning -- cow path, Madsen urged against what might be called path-of-least-resistance BI. Yes, he concedes, the cowpath is already there. Yes, it's well-understood; yes, it's functional; and, yes, it goes (eventually) where you need it to go. However, this doesn't mean it's the shortest, most direct, or most advantageous route, he stressed.
That, in a nutshell, is the argument for insurgent BI. Although BI suites aim to appeal to both IT and the line of business, insurgent BI offerings are designed primarily with end users in mind. Insurgent BI likewise makes virtues out of its most egregious liabilities: in the case of a vendor such as WhereScape, this could be its perceived immaturity -- i.e., its lack of fit and polish relative, at least, to full-fledged data integration (DI) offerings from suite vendors such as IBM Corp., Oracle Corp., SAP AG, or SAS Institute Inc. A vendor such as Bis2, on the other hand, likes to champion an unconventional approach to data visualization that a data visualization purist might find objectionable.
In every case, insurgent BI emphasizes empowerment -- by virtue of, for example, rapid-time-to-deployment DI (in the case of ETL tools such as WhereScape's RED or Expressor 3.0); analytics optimized for both user self-service and collaboration (in the cases of Lyza and QlikTech); or data visualization that distorts -- in some cases, critics allege, drastically -- in order to enlighten. (This last category was effectively inaugurated by Bis2.)
What's intriguing is that the once-modest revival of "workgroup BI" has developed into an irksome -- if still far from momentous -- market force.
Insurgent BI bears watching in 2011.
Social Studies
In 2010, BI -- like just about everything else -- got a little bit more sociable.
LyzaSoft kick-started the revolution in February, at TDWI's World Conference in Las Vegas, unveiling a Lyza 2.0 release that incorporates social media concepts and methods -- including searching; matching and combining; tagging; sharing; commenting; and rating -- into an analytic context.
Industry watcher Ted Cuzzillo argued that Lyza 2.0 stole the show at Las Vegas TDWI. He might've been on to something: Lyza's social media retrofit had conference attendees, third-party vendors, and industry analysts buzzing.
Not all of this buzzing was salutary, of course. Fact is, Lyza's push into social BI begs those most timeless of questions: wherefore and what for?
Ten months later, many still can't agree on the wherefore (to say nothing of the what for). Advocates argue that social networking concepts and methods are logically consistent with the evolving BI user experience; skeptics point to social networking concepts and methods that -- while likely of some value -- could prove to be more distracting and productivity-killing than beneficial.
Not surprisingly, the push to incorporate social networking concepts and methods into BI or DW likewise reprises a familiar theme: that of insurgent BI.
In social BI, users are ascendant, argues LyzaSoft CEO Scott Davis; as a result, IT and DM stakeholders -- who have traditionally hoarded power -- are going to have to relinquish some of their most cherished beliefs.
"You have to be realistic about it. You're not going to be able to impose a priori that sort of top-down model where you control everything centrally," Davis said. "It's sort of turning everything upside down. You're saying that instead [of IT], the users become the incubation zone for new ideas and IT comes along and picks the best of them and migrates them into different places."
As of December, Lyza is probably still the most sociable of BI tools. The rest of the industry has had nearly 12 months to take stock, however.
If Davis is right, and if social BI is inevitable, then any BI player worth its salt is going to have to have a social strategy. In fact, 2011 could be the year in which BI gets a top-to-bottom social retrofit. How well it fits anybody's guess.
Going Mobile
In April, Apple Corp. introduced the iPad, a category-re-defining deliverable that -- in the eyes of a many -- recast the PC as bloated and bulky. Compared to the sleek and stylish iPad, the thicker, bulkier Tablet PC -- to say nothing of the cumbersome end-user desktop -- seem downright inconvenient.
BI vendors were quick to adopt the iPad as a next-generation client device, and although it's far too early to say that the iPad has affected BI, it seems likely that mobility will transform both BI and DW. In this regard, an iPad-like device could become the killer medium for mobile BI. After all, an iPad-like interface -- with touch-screen interactivity and vivid LED-backlit display -- shows visual elements such as dashboards to great effect.
The list of BI types that are supporting the iPad includes prominent analytics vendors (e.g., QlikView, SAP BusinessObjects, MicroStrategy Inc.); best-of-breed data visualization players (TIBCO Spotfire); and CRM vendors (salesforce.com and SugarCRM). At least one vendor -- MeLLmo Inc., developer of Roambi -- markets BI-oriented reporting, analysis, and (limited) visualization offerings specifically for the iPad and iPhone.
With Samsung, Research in Motion Inc., and other vendors unveiling iPad-like devices of their own -- and with industry watcher IDC projecting that the combined base of smartphones and tablet computers will surpass that of the PC by 2012 -- look for mobile BI to have a huge impact in 2011.
As for the meaningful intersection of mobile BI and social BI, let's circle back next December. We could be publishing a special iPad-only edition by then.