Grading BI Vendors
A new research tool could fill a gaping void in the BI-osphere.
- By Stephen Swoyer
- June 9, 2015
The Business Applications Research Center (BARC) is perhaps best known for its business intelligence-oriented publications -- starting with its sweeping BI Survey, now in its 15th edition.
BARC recently published a new research report, BIScore, that rates BI vendors on a variety of criteria, from the capabilities of their products to their sales and marketing strategies. BARC's BIScore metric also weighs both a vendor's financial performance and feedback from customers. Think of it as a hybrid of something like Gartner Inc.'s Magic Quadrant and the BI platform evaluations that Cindi Howson (now a Gartner analyst) used to provide at BIScorecard.com.
It's possible that BARC's BIScore could fill a gaping void in the BI-osphere. For one thing, it assesses products on the basis of BARC's own criteria, but (like Howson's sorely missed BIScorecard.com) it also gives priority to customer experiences. For example, BARC says its BIScore vendor ratings incorporate product satisfaction metrics from its BI Survey; theoretically, this gives BIScore an empirical basis that's facilely comparable to that of BIScorecard.com, which -- prior to Howson's transition to Gartner -- used to publish the hugely informative Successful BI Survey.
On the product side, BIScore isn't simply a tally of bells, whistles, and platform arcana. According to BARC, the vendor ratings give priority to products with strong dashboard and ad hoc query capabilities; they also accord importance to the strength of a vendor's architecture and (thanks to the BI Survey data) to overall customer satisfaction; other features or experiences (such as vanilla "reporting" and "data mining") are less heavily weighted. As for the not-so-tangible measures, BARC says BIScore gives more weight to a vendor's product strategy and strength of customer support than to other factors -- including sales support, marketing strategy, or organizational strength.
Who came out on top?
It's hard to say. Like Gartner's Magic Quadrant, BARC plots vendors on a vertical "Y" axis (which corresponds to "Market Execution") and a horizontal "X" axis, or "Portfolio Capabilities."
The inaugural BIScore has IBM Corp. and SAP AG receiving top marks for market execution (with IBM scoring highest, followed by SAP), but trailing MicroStrategy Corp. and Information Builders Inc., respectively, in portfolio capabilities. BIScore places both MicroStrategy and IBI as #1 and #2 (respectively) in "portfolio capabilities." Fine, you say, but what does this actually mean?
That's the trick. BIScore doesn't aspire to provide a categorical ranking of BI vendors, i.e., on the basis of their weighted strengths and weaknesses, from top to bottom. It's instead a synthetic score, which -- however potentially frustrating to would-be buyers of BI tools -- is a good all-around compromise. No single BI platform can be all things to all customers, in spite of what BI vendors would have you believe. BIScore provides a breakdown of each vendor's strengths and weaknesses -- it features an analysis section with detailed run-downs on each vendor.
Take IBM, for example. BARC evaluated Big Blue's Cognos BI and TM1 (the erstwhile Applix in-memory OLAP engine), along with its Watson Analytics and SPSS assets. BARC's BIScore notes its strengths in formatted and ad hoc reporting, analysis, dashboarding, and planning, as well as its "broad capabilities for data mining and advanced analysis." This encompasses the "creation and deployment of individual models as well as [the availability of] business-user-oriented encapsulated capabilities," according to BARC. The BIScore rating also cites IBM's Watson Analytics offering -- which Big Blue shipped just last year -- as an "innovation leader." Furthermore, because BARC's BIScore rating gives more weight to the strength of a vendor's ad hoc query and analysis capabilities, IBM scores highly in that regard, too. Not only does it have TM1 -- traditionally one of the fastest OLAP engines in the business -- but it also has its Cognos BI Workspace Advanced.
On the other hand, IBM -- like most of its competitors -- cobbled together its BI stack by dint of a series of acquisitions, gobbling up vendors such as AlphaBlox, Applix, Cognos, and SPSS, among others. One result of this (even eight years on in the case of Cognos and six years on in the case of SPSS) is "limited integration between ... product families [such as] IBM Cognos BI, IBM Cognos TM1, IBM SPSS and IBM Watson Analytics," according to BARC.
Big Blue also gets dinged for its "potentially confusing mobile BI strategy due to multiple products in the portfolio" -- e.g., IBM Cognos Mobile, IBM TM1 Mobile Contributor, and IBM Concert Mobile. AlthoughWatson Analytics was a signal deliverable, it isn't yet clear how IBM will reconcile that product with other extant offerings in which there's overlap. "With Watson Analytics being IBM's strategic data discovery offering, there is limited traction of personal analytics solutions Cognos Insight and its AnalyticsZone community."
Here's another example: BIScore's rundown of MicroStrategy, which scores highest in terms of the overall strength of its portfolio. "MicroStrategy has one of the best architected suites on the market, and its integration of Flash into the system is particularly impressive. MicroStrategy centrally integrates all BI functions in the suite instead of offering a stand-alone self-service BI client," BARC writes.
Disregarding BARC's seemingly strange praise of Adobe Flash -- the market prefers the non-proprietary HTML5 standard, even if BI players haven't caught up -- this jibes with BIScore's tally of MicroStrategy's strengths: viz., it offers a single "integrated platform for formatted and ad-hoc reporting, dashboarding, analysis and BI application creation with good performance in large and complex environments." Similarly, MicroStrategy's Visual Insight -- which first shipped three years ago -- "offers capabilities for visual and explorative data analysis and addresses current requirements in the self-service BI area supporting the integration of data as diverse as local spreadsheets or Hadoop." MicroStrategy also gets high marks for its cloud platform, which analysts such as Howson have also applauded.
MicroStrategy has its weaknesses, starting with limited performance management facilities and a complete lack of planning capabilities. MicroStrategy also lacks a product that's dedicated solely to predictive analysis and data mining, according to BARC. At this point, it supports only predefined models. Although a company's doings and shakings aren't always important or apposite, BARC's BIScore rating does ding MicroStrategy for an "extensive" reorganization that included the reshuffling of executives and staff layoffs. Finally, even though MicroStrategy scores highest in terms of portfolio capabilities, all of those features and functions come at a high price: BARC is simply confirming conventional wisdom when it notes that MicroStrategy typically has higher licensing costs than its competitors.
BARC's BIScore rating isn't the last word in BI. It can't be and it doesn't pretend to be. It does, however, have the makings of a good start, insofar as it offers more substantive detail (and, thanks to its inclusion of product satisfaction data from BARC's BI Survey, a critical customer feedback loop) than is available via other (extant) research resources. BIScore can't and shouldn't be used as a tool on which solely to base a BI platform buying decision. Its ideal use is as a tool to supplement such a decision. In this respect, it can and does fill a void in the BI market.