Perspectives: Open Source Business Intelligence's Very Bright Future
Some issues that complicate or frustrate shrink-wrapped software implementations are mitigated by the open source model. That has some experts predicting a very bright future for open source BI.
- By Stephen Swoyer
- December 9, 2009
Market watchers say free and open source software (F/OSS) business intelligence (BI) and data warehousing (DW) solutions are fast emerging as strong forces to be reckoned with. What's more, as open source BI and DW offerings mature, it's possible they'll comprise an ever-larger share of enterprise software evaluations. After all, some of the very issues that work to complicate or otherwise frustrate commercial off-the-shelf software (COTS) implementations are mitigated by (or non-operative in) the open source model. The upshot, experts predict, is a very bright future for F/OSS BI and DW.
To be sure, the success of any F/OSS business intelligence or data warehousing project -- like that of a project based on a commercial offering -- is by no means a slam dunk. This shouldn't be surprising because a significant percent of IT projects fail. ERP project failures are perhaps the most notorious in this regard. According to one tally (published in 2001 by management consulting services specialist Robbins-Gioia), more than half of adopters considered their ERP implementations to be failures. Elsewhere, a recent Standish Group survey found that a majority -- 68 percent -- of IT projects are judged as something less than successful, with only 32 percent being declared clear-cut "successes."
Failure rates for build-your-own software efforts are (depending on the research methodology you trust) as bad -- or worse. For example, a 2005 study from analyst think tank The Cutter Consortium found that nearly half of shops had abandoned one or more build-your-own software projects at some point during the survey period.
Would-be adopters try and fail with open source BI and DW solutions, too. The difference lies in just why and when they give up on F/OSS tools.
More to the point, a sizeable percentage of potential adopters reject open source BI and DW entries during the evaluation phase: nearly half (47 percent) of respondents in a F/OSS survey conducted by consultancy Third Nature reported at least one failed open source BI or DW evaluation.
To a degree, this is a function of the relative immaturity of F/OSS business intelligence and data warehousing solutions: would-be adopters are able to determine early on that a specific offering (or -- as is increasingly the case -- bundle of offerings) just isn't mature enough to do what they need it to do.
In other cases, the Third Nature survey indicates, shops determine that the cost of switching to open source BI or DW tools -- chiefly as a function of re-writing existing reports or integration jobs, or retraining in-house talent to exploit open source technologies or methods -- is prohibitively expensive.
The salient point, notes veteran DW architect Mark Madsen, a principal with Third Nature, is that potential adopters are able to quickly and conclusively determine whether open source BI or DW tools are a good fit. In the vast majority of cases, Madsen indicates, they're able to do as much during the evaluation process itself. Thanks both to its download-and-deploy mechanics and its built-in transparency -- the "open" in "open source" -- evaluating open source BI and DW tools is easier (in that assessments are both more conclusive and more predictive of eventual success or failure) than is evaluating commercial or build-your-own tools.
What's more, as open source business intelligence and data warehousing tools continue to mature, and as commercial open source software (COSS) vendors continue to tackle some of the perceived shortcomings of the F/OSS BI and DW models, open source tools will likely be evaluated in an increasing number of situations.
"Software maturity is the primary limitation," Madsen writes. "This includes problems like scalability and reliability, with 'missing required features' at the top of the list. The single biggest write-in complaint was poor documentation, something that F/OSS projects often struggle with. This is one of the gaps COSS vendors are trying to fill to make the software more enterprise-friendly."
To be sure, not all of these evaluations will result in adoptions; moreover, it's likely that some fixed percentage of open source BI and DW project implementations will -- like some fixed percentage of commercial off-the-shelf (COTS) and build-your-own project efforts -- be deemed failures. That being said, BI and DW projects based on F/OSS or COSS assets have several built-in advantages relative to their COTS and build-your-own counterparts, experts argue.
"Gone are the days when the business would give IT some leeway to see if something 'might work,'" write industry luminaries Claudia Imhoff and Colin White in "Open Sesame," a recent open source BI and DW prospectus. "Testing BI technologies or potential applications was a useful way for IT to learn what worked and what didn't, but it was also fairly expensive and led to disillusionment [among] business users," they continue, contrasting the limited configurability and testability of COTS offerings with the download-and-deploy model popular in the F/OSS and (via so-called Community Editions) COSS segments.
"The ability to test whether a technology might work was quite limited; implementers either had to buy the technology outright or get a limited-time license which may have been too short for a proper trial."
The perverse upshot, then, is that -- owing to the download-and-deploy spirit of the open source model -- F/OSS BI and DW offerings aren't just likely but will inevitability to be included in project evaluations.
"One of the advantages of open source previously cited was the ability to quickly and easily download software and get started -- with no upfront cost," write Imhoff and White, who suggest that large enterprise shops are already ripping out COTS offerings and replacing them with F/OSS or COSS tools "where it makes sense" to do so. In other words, when shops decide they're comfortable with the maturity and support levels of F/OSS or COSS tools, the built-in advantages of the open source model (download-and-deploy test-drivability; extensive configurability; absence of vendor lock-in) cinch the deal.
Madsen, Imhoff, White, and other experts point to the instability (in Madsen's view, the egregious unsustainability) of COTS licensing schemes, which frustrate, bind, and (in many cases) alienate customers.
One reason for the growth in OSS BI and DW offerings -- and one reason these solutions are (or soon will be) included by default in a majority of software evaluations -- is that they offer choice, chiefly with respect to customizability, flexible licensing terms, and (for shops that want or need to extensively tweak their BI or DW implementations) thriving community support.
"One of the primary factors driving customer adoption has been a rebellion of sorts against vendor lock-in or the payment of a software license," write Imhoff and White. "The combination of a lower cost of production with the freedom and flexibility prevalent in open source deployments has been driving both adoption and production," they continue. "The yin of BI software commoditization and the yang of customer demands for lowered costs of ownership have created a perfect opportunity for open source software and the open source community."
For these reasons and others, Third Nature's Madsen, who -- in a July presentation to the Boston Chapter of TDWI famously forecast the "breakdown" of the enterprise software market -- urges that open source BI and DW offerings should be included as "default" options in certain kinds of software evaluations.
"When in an environment with few or no tools, open source should be the preferred alternative," writes Madsen. In such shops -- where inertia (in the form of switching costs or built-in vendor-specific expertise) doesn't act as a switching restraint -- open source is a no-brainer default, he argues. "It is the simplest, fastest, and likely the least-expensive route when compared to hand-coding or proprietary products."
In shops with COTS BI or DW investments, Madsen suggests, open source solutions are in many cases mature enough that they merit consideration -- not, perhaps, as rip-and-replace but as supplementary propositions -- in the evaluation process. "One of the obstacles encountered while evaluating open source BI or ETL tools was the high cost of redeveloping reports or integration jobs," he points out. "Rather than look[ing to save] money by replacing software, look at gaps in the BI portfolio or data warehouse stack and use open source to supplement your systems."