TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

Celebrating Self-Service

In the New Information Economy, self-service use provides an empirical basis for demand. If people repeatedly use self-service tools to construct certain kinds of data flows, to perform certain kinds of analyses, or to access and integrate data from as-yet-unmanaged internal or external data sources, that's a critical signal to IT. These are use cases that must and should be identified, standardized, and productized as reusable information assets.

• By Steve Swoyer
• February 22, 2016

There's no polite way to put this: self-service business intelligence (BI) and data preparation tools used to scare data management (DM) practitioners.

As the DM establishment saw it, self-service tools reinforce an isolated, one-off work experience. They aren't designed for (and in virtually all respects do not promote) the standardization and reuse -- the productizing -- of analytical insights and data flows in operational processes.

Worst of all, DM practitioners groused, they're ungovernable. This was the IT equivalent of the proverbial pot calling its counterpart -- the kettle -- black.

If self-service tools reinforce an isolated, one-off work experience, the data warehouse reinforces a no less isolated, no less one-off work experience. This has very little to do with the design and conceptual architecture of the data warehouse and almost everything to do with the way DM practitioners built, managed, and governed warehouse systems in practice.

The data warehouse was conceived as a pragmatic platform for information delivery and management. It was designed to support (and, in some cases, to enable) business decision-making. In practice, however, DM practitioners emphasized their own priorities -- reuse, repeatability, and a restrictive governance regimen -- and gave short shrift to those of the business constituencies they were charged with serving. In effect, they created a restrictive culture of control. As far as many business people were concerned, the data warehouse discouraged use; was an impediment to information delivery, analysis, and decision making; and was a drag on productivity.

In other words, the data warehouse got in the way of people doing their jobs. This is one reason spreadmarts have been with us for as long (and longer) than the data warehouse itself. It's why the advent of self-service in the early and mid-2000s found an eager, built-in audience in the enterprise. (The first self-service tools were basically OLAP tools with in-memory, e.g., Qlik, or data visualization -- Spotfire, Tableau -- features.) A sizable proportion of the people for whom the data warehouse was designed, built, and operated weren't using it -- or weren't willingly using it.

They weren't willingly using it because the data warehouse-driven BI model didn't address their needs. It didn't let them do what they either wanted or needed to do. It stymied them instead of supporting and enabling them.

A 1981 episode of the BBC sitcom Yes Minister neatly captures this problem. It depicts a London hospital that's staffed by 342 administrative staff and 170 support staff. This same hospital has zero medical staff and zero patients. "Isn't it appalling that it's not being used?" asks Jim Hacker, the show's titular Minister for Administrative Affairs. "Oh no," the hospital's administrator says. "[It's] a very good thing in some ways. Prolongs its life." It doesn't matter that there aren't any doctors, nurses, or patients, she tells him. "The essential work of the hospital still has to go on." When Hacker objects, the administrator protests: "But Minister, it's one of the best-run hospitals in the country!"

It's easy to see this as a biting critique of governance and manageability policies that are so restrictive as to preclude the effective use of the data warehouse by the very people ("users," in IT-speak) for whom it was designed, funded, and built. There's undeniable truth to this, too. DM practitioners have, indeed, been overly restrictive -- irresponsibly restrictive, perhaps -- in building, managing, and governing the data warehouse. This is changing, however.

In many ways, it already has changed. DM teams, BI vendors, even vendors that market high-performance data warehouse and analytic systems. All emphasize the need for their technologies to be agile, adaptable, flexible, and responsive. This change has a lot to do with unrelenting pressure from self-service vendors and self-serving users. Marc Demarest, a research analyst with information management consultancy Noumenal Inc., likes to describe the old data warehouse-driven BI model as a Soviet-style "planned information economy." In this scheme, Demarest argues, demand was completely decoupled from production.

This doesn't mean demand wasn't being serviced, however; before we had neo-self-service tools, Demarest notes, we had the spreadmart, the original DIY technology. "Even highly corrupted fourth-generation versions of [data extracts], stuff you don't know where it came from: those extracts are the traces of demand. If you frame that as a failed supply problem, you recognize that demand will always be met, one way or another," he argues.

"Every moment that your smart people spend doing extracts for other people because the data warehouse sucks ... is a moment they're not spending on productivity."

In retrospect, the spreadmart was a kind of metaphorical samizdat to the planned information economy of the data warehouse-driven BI model. During the Cold War, "Samizdat" described the production and dissemination of censored publications by dissidents in the former Soviet Union and throughout the Warsaw Pact. As a phenomenon, samizdat was indicative of a kind of unacknowledged demand -- in this case, for proscribed materials. Its existence also hinted at the inability of Soviet ideology to meet the needs of all citizens.

As Demarest sees it, self-service is playing a similar, albeit ongoing role, first as a corrective to a broken production-driven model -- the Old Information Economy -- and, second, as a kind of demand signaler in the New Information Economy. In the former instance, the existence and growth of self-service technologies demonstrated to IT (as well as to business executives) "You aren't meeting our needs. You haven't been meeting our needs. We're going to go around you."

Self-service was a dissident movement that first put pressure on and (in combination with other social and economic forces) finally toppled the Old Information Economy.

In the New Information Economy, self-service use provides an empirical basis for demand. If people repeatedly use self-service tools to construct certain kinds of data flows, to perform certain kinds of analyses, or to access and integrate data from as-yet-unmanaged internal or external data sources, that's a critical signal to IT -- and to the data warehouse team. These are use cases that must and should be identified, standardized, and productized as reusable information assets.

Demarest sees self-service as a vital signaling mechanism. "Data-driven decision-making cultures are about balancing supply and demand properly in a kind of free-market economy. If you look at it critically, BI is just a distribution mechanism; it adds no value, except maybe logistical value," he says. "The irony is that BI was an electronic version of the prior distribution system. A kid with a roll-y cart shows up at your office every Tuesday and plops a greenbar report into your 'In' tray. BI was basically an attempt to take that primitive infrastructure and replicate it with information systems. Why would you want to do that?"