BI/DW: The Year Past, The Year Ahead
Self-service and agility were hot topics for BI users in 2012. What does 2013 hold?
By Michael Whitehead
Two things were as true in 2012 as they were 25 years ago. First, data warehouses, then as now, are too hard to build and too hard to change. Like petroleum supertankers, they're slow, monolithic, and hard to steer. Once you have them pointed in a certain direction, it's easier to keep on going because it takes too long to turn them around again.
Second, business intelligence tools might look a lot better than they did 25 years ago, but they're only marginally more useful. Getting information out of a BI tool can be maddeningly counter-productive. BI tools tend to be slow, hard to use, and inflexible.
Much of what happened in BI and data warehousing over the last 12 months focused on these problems. Much of what we're predicting for 2013 will too.
What's different this time around is hope: the industry seems more serious about tackling some of BI's most intractable issues.
With that, let's plunge into our prospectus.
2012 Trend #1: Big data went boom
The first half of the year gave us the "3Vs:" volume, variety, and velocity. Thankfully, the industry got past this. We realized that the 3Vs say nothing about why big data is special. Instead, we started framing big data in terms of its value or transformative potential. Take big data-infused supply chain management: it has the potential to proactively identify problems – e.g., disruptions caused by earthquakes, flooding, geopolitical flare-ups, labor or work stoppages, and so on – and to recommend solutions. If it can be realized, it's a game-changer. Over the course of 2012, vendors grew to embrace the idea, but will customers follow, and can big data live up to its so-hyped potential?
2012 Trend #2: BI wanted to become more agile
In 2012, there was genuine enthusiasm for agile concepts and methods as an antidote for "broken" BI. TDWI's Boston World Conference focused on "agile BI," as did several other industry events, including a seminar at this summer's Pacific Northwest BI Summit.
(Full disclosure: WhereScape supports agile development. Many of our customers use WhereScape RED to drive agile development practices.)
For most of its existence, agile's been seen as inescapably methodological. That's starting to change. For purposes both practical and self-serving, the BI industry is promoting agile on the basis of its agility. At TDWI's "Agile BI" conference, everyone from Microsoft to Teradata was talking agile; in most cases, they didn't mean a methodology such as scrum.
This is as it should be. "Agile" is a matter of practices, not strictly an issue of methodology. You can do agile without actually being agile: you can use agile concepts, but if you're hamstrung by the too-rigorous application of a methodology, you're the furthest thing from "agile."
If you're working with a data warehouse, this means having the flexibility to rapidly generate a new schema, or to discard an existing schema once it becomes a barrier to agility. It means using tools to automate as many routine tasks as possible: things such as error handling, status updates, documentation, or naming conventions.
2012 Trend #3: BI users were keen on self-service
BI tools such as data warehouses have always been hard to deploy and even harder to use.
This year, the industry seized on a new hope for fixing what's wrong with BI: self-service discovery. It started a few years ago, when business users started bypassing IT to buy and implement "discovery" tools such as Tableau or QlikView. These tools promised to minimize IT's role in implementing and supporting them. Over the last 18 months, the industry produced a slew of "copycat" offerings: self-described "discovery" tools (such as Visual Analytics Explorer from SAP BusinessObjects, or Visual Intelligence from SAS, which appeared in 2012) that aim to empower users to do some of the things that used to devolve upon IT.
A Look Ahead
Here are my first predictions for BI and data warehousing in 2013.
2013 Prediction #1: BI becomes more self-serviceable (but still isn't self-serviceable enough)
The industry hopes that by building self-service capabilities into BI tools and by co-opting the metaphor of discovery, it can make BI more responsive. It can put users, not IT, in control.
It isn't going to happen. If self-service discovery is used as a front-end for the same inflexible, hard-to-change data warehouses, it's bound to fail.
If it's used by itself, in the classic QlikView or Tableau model, it's also bound to fail -- on the enterprise level, at least. This means accepting an Inconvenient Truth: BI just can't be made completely self-serviceable. Data access will continue to be a problem. (If a traditional data warehouse is involved, it will be a big problem.) Before a user starts exploring with a discovery tool, someone has to provision and prepare their data for them. Even vendors such as QlikTech or Tableau, which became successful because they eschewed sophisticated data preparation, have come around: QlikTech by buying an ETL vendor to address its data integration shortcomings; Tableau by incorporating enterprise amenities into its more recent releases. Self-service discovery isn't a panacea.
2013 Prediction #2: Data virtualization goes back to basics but with an agile twist
An agile take on data warehouse design will trump newfangled alternatives. These include self-service discovery and data virtualization (DV), both of which purport to address the "problems" (chiefly, an inflexible data warehouse) and "bottlenecks" (IT's supporting role) that bedevil BI.
Unlike self-service discovery, DV doesn't propose to do away with IT: someone still has to build and manage the canonical views (that power its dashboards) and composite applications. This takes time, and (like traditional ETL development) doesn't happen overnight – or in a couple of weeks. What's more, IT has to build unique views for each of the roles or applications a business wants to enable. In doing so, IT is hampered by the connectivity limitations of the underlying DV. Think of a DV as a kind of Swiss Army knife: there are applications for which its scissors or screwdriver tools are acceptable. They're never ideal, however.
An agile approach to building data warehouses is ideal. It addresses the Achilles' heel of self-service BI; it's a superior alternative to the monolithic data warehouse; it's more flexible in practice than a DV – and easier to build out, too.
It leverages intelligent automation where it makes sense to do so. You start by prototyping; once you start building, you don't obsess about whether you have everything completely right -- e.g., whether you've defined all possible requirements or anticipated all possible use cases. You use tools that automate most of the tasks you'd otherwise find yourself doing repeatedly. The point is that you're able to rapidly build application-specific data marts or data warehouses. It's an iterative process that evolves -- very rapidly -- over time. If business conditions change, or if your company makes a large acquisition, you aren't hamstrung by design decisions that you (or someone else) made a decade ago.
2013 Prediction #3: Data warehouses are as popular as ever
From data virtualization to in-memory analytics to self-service discovery, there's no shortage of "fixes" for what's wrong with the data warehouse. None of these is transformational enough to topple it. Over the years, many technologies have been touted as "data warehouse killers." The data warehouse has survived, however.
This is because the data warehouse is the archetypal decision-support platform. When you move from a discovery platform to a decision platform, you're applying a different set of disciplines. You want systems that are easy to maintain and that reliably deliver the same types of information. You want to be able to apply business rules in one location; before you know it, you're persisting data. You want valid comparisons, so you need to ensure that you're staging your real-time/right-time/hourly data in a consistent and reproducible manner.
For these and other reasons, the current architectural approach is pretty much what you want; the concept of the data warehouse is sound. Now as ever, it's the preeminent platform for decision support.
Michael Whitehead is the founder and CEO of WhereScape Software. He has been involved in data warehousing and business intelligence for more than 15 years, and is a strong proponent of value based data warehousing and data warehouse automation. Michael has spoken at numerous conferences worldwide on the topics of agile data warehousing and data warehouse automation. You can contact the author at [email protected].