How Agile Requirements- Gathering Can Conquer Analysis Paralysis

Michael A. Schiff
Principal Consultant, MAS Strategies

Topic: Experts in Data Warehousing

We live in an imperfect world, yet some people still believe that if something can't be done perfectly, it shouldn't be attempted at all. Although this may be an extreme view and may be totally appropriate in some situations (such as building a nuclear power plant or performing surgery), it should not be a showstopper for data warehouse implementations.

How many times have you encountered situations where a feasibility study of a major system (operational or analytic) continually missed deadlines because of the supposed need for additional research and analysis? Furthermore, when the analysis was completed (or management insisted that the project go ahead without additional delay), the implementation date almost always passed the initial estimate.

This phenomenon is sometimes referred to as "analysis paralysis," and I have personally observed it all too many times. I still remember an incident when a marketing director, when estimating product demand, delayed an entire team's associated efforts as he continually recomputed his numbers. We finally realized that he was attempting to estimate percentage demand across channels down to two decimal places even though he was instructed that management wanted "ballpark" estimates within 5 percent. He (wrongfully) thought that his overly cautious attitude would reduce the possibility of an error and was oblivious to the fact that by the time his analysis was completed (if ever) and decisions made, any associated benefits might be long gone.

Data warehouse practitioners need to remember that before the term business intelligence was coined, organizations built what was known as a decision support system (DSS) facilitate and support better decisions. Because a primary goal of a data warehouse is to support business intelligence and make information available to those who need it when (and where) they need it, we cannot get bogged down by analysis paralysis. Instead, we must strive for agility and create an environment that allows our users to receive timely, relevant information.

We need to remember that there are costs associated with delayed decisions and that once an opportunity is lost, it may never be available again. We have all experienced situations where users' initial specifications didn't match actual user needs, even after formal functional specifications were approved. In most cases, the specifications are a starting point that usually lead to requests for additional details, additional data, and/or other report formats.

Analysis is frequently an iterative process and it is safe to assume that most answers will lead to additional questions; questions that were not thought of until the prior questions were answered. "What about..." may begin the most frequent user comments we hear after users are shown initial results.

One approach to facilitating an agile data warehousing and business intelligence environment that deserves special attention is prototyping. It can help us ensure that our reports and analyses are actually what our users need -- which may or may not be what they initially though they wanted!

Although sometimes the ability to prototype is constrained because the data we need does not already reside in our data warehouse, we can use techniques such as data virtualization or enterprise information integration (EII) to access this data from operational systems. Although the data in the operational systems may not be of the same quality as data formally loaded into a data warehouse, the approach can provide a "quick-and-(sometimes very)-dirty" solution. If satisfactory, the data can ultimately be formally cleansed and included in the data warehouse.

Even if our initial analysis provided all the answers our users needed, their requirements and associated queries will evolve over time. One of the best ways to deal with this is to design our data warehousing and business intelligence applications with agility as a major objective and not get bogged down by analysis paralysis in an attempt to perfectly solve initial requests that may (or may not) reflect what our users really need.

Michael A. Schiff is a principal consultant for MAS Strategies. He can be reached at mschiff@mas- strategies.com.