Big Data from a BI View
Big data developments unveiled at sold-out Strata Conference in New York.
By Cindi Howson, BI Scorecard
I was lured to Strata by both the traditional vendors I cover (such as SAP, SAS, and Tableau) exhibiting there, as well as the big data analytics start-ups such as Datameer and Karmasphere. There's a high degree of hype around big data, but there's also a high degree of innovation, tangible benefits, and venture capital.
Coming from the BI world, here's the skinny on where big data meets BI.
First, big data is more than Hadoop, the open source distributed file system able to scale to petabytes of data. The ability to handle high volumes of data is not the only appeal to Hadoop; it's also that it can handle unstructured data such as clicks, tweets, video, Facebook comments, sensor data, and so on. Such data is a challenge to model and store in a traditional relational database and data warehouse schema. When I first suggested to a fellow attendee that big data is more than Hadoop, he warned me such a comment might be considered blasphemous at Strata.
In the traditional BI world, technologies such as analytic appliances, columnar databases, and in-memory engines can also handle big data. It all depends on whether the challenge is volume and performance, variety and complexity, or combinations thereof.
Which brings me to the first big announcement at the conference: Cloudera Impala, a new real-time query engine for Hadoop. Cloudera is the leading provider of Hadoop software and support. It competes with Hortonworks and MapR, among others. In the last year, many BI vendors have announced support for Hadoop. Access to data in Hadoop has been through Hive, a virtual data warehouse for Hadoop that has its own query language, HiveQL. The thing is, HiveQL generates MapReduce jobs to get to the data in Hadoop. MapReduce is batch-oriented and slow, in contrast to BI, which is supposed to be fast -- but hey, we might be talking petabytes here, so maybe a little patience is reasonable.
Impala, in contrast to Hive, is real time. With Cloudera leading the Impala development for the past two years, early adopters wondered if the technology would be kept proprietary to Cloudera. Instead, Cloudera says it will release it to the open source community. The first BI vendors to support Impala are MicroStrategy, Tableau, Pentaho, and QlikView. Pentaho cited a 10x query performance improvement using Impala over Hive, but acknowledges it's still not as fast as what users are used to in the BI world. On the high end, Tableau has seen up to a 100x performance improvement with dense tables of numbers, so performance improvement depends on the type and structure of the data in Hadoop.
What's an impatient BI user to do? This is where the rest of the big data architecture comes into play. Querying vast volumes of granular data in Hadoop via Hive may be slow, so once an initial exploration is done, BI vendors will cache the results in their technology to ensure speed-of-thought analysis. This is where solutions such as SAP Hana, MicroStrategy OLAP Services (for in-memory), SAS LASR server, the Tableau Data Engine, or the QlikView in-memory engine all come into play. Startup vendor SiSense was also at Strata, showcasing its combination columnar and in-memory engine, exploring 1 terabyte of data on a simple laptop with only 8 gigabytes of RAM.
With Hadoop gaining traction, there are a number of new data visualization and exploration vendors that offer access to and exploration of data in Hadoop. Isn't that what BI vendors with Hadoop connectors do? Yes, but traditional BI vendors can also access data in a data warehouse, analytic appliance, or spreadsheet. Hadoop may or may not be part of the picture.
With the new big-data-oriented vendors, such as Datameer, Karmasphere, and Platfora, Hadoop is definitely in the picture and may be the whole picture. Datameer, for example, generates its own MapReduce jobs (not relying on Hive) and provides 40+ connectors to sources such as Salesforce, Twitter, and Google Ads that allow you to load data into Hadoop. Initial data is explored through a spreadsheet-style interface and then cached in Datameer's own in-memory engine. At the conference, Datameer announced a new app market with 30 initial apps that are free.
Karmasphere takes a different approach, focusing on data access and query generation through Hive. It partners with Tableau for visualization. Platfora, currently in beta, has an appealing front-end that reminds me of MicroStrategy's new Visual Insight. It queries Hadoop through MapReduce jobs to create what it refers to as a lens. Lenses are in-memory caches of the data sets that users can then visually explore. The product is expected to become generally available in Q1 2013.
It was refreshing for me to see a mixture of data scientists and technologists at this conference. Both camps seem to recognize that the explosion of data along with the shortage of talent in this space is creating a perfect storm in which only the smartest will survive. Survival of the smartest has been prevalent in the recession. Some companies are still struggling to analyze sales and who's buying what. In the big data economy, the analysis extends to who's interested in your products, who's influencing buying decisions, and who's not even engaged but should be!
Cindi Howson is the founder of BI Scorecard, an independent analyst firm that advises companies on BI tool strategies and offers in-depth business intelligence product reviews.