Making the Case for Agile Business Intelligence
A new report from Forrester Research makes the case for BI agility. The bottom line: business intelligence can -- and must -- become more responsive.
- By Stephen Swoyer
- May 5, 2010
A decade after its insurgency, agile programming is an established (and in most cases, respected) approach to application development. Now there's a push to assimilate agile concepts and methods into business intelligence (BI) and data warehousing (DW) practices.
The forthcoming TDWI Summer World Conference in San Diego, for example, has Agile BI as one of its major themes. Meanwhile, some BI and DW vendors -- including search applications specialist Endeca Technologies Inc., which recently commissioned an "Agile BI" study by Forrester Research -- tout a notionally "agile" approach to BI or DW as a means to differentiate their products.
One impetus, as always, is lagging BI usage, such as survey data indicating that BI uptake has basically flat-lined. Some BI professionals, concerned about the ability of enterprise data management (DM) teams to respond to the rapid pace of business change, tout the adoption of agile concepts and methods as one way to increase the responsiveness of BI and DW.
When agile programming first appeared, it advocated seemingly heretical application development methodology: in place of time-consuming background research -- e.g., a lengthy and exhaustive requirements analysis -- onerous documentation, and a protracted, pre-testing development cycle, agile championed a develop-as-you-go approach. Agile advocates like to emphasize the importance of early (and frequent) prototyping -- there's even an agile offshoot called "test-driven development" -- as well as ongoing collaboration between users, business stakeholders, and developers. Advocates claimed that agile could both accelerate the pace of application development and deliver more usable, functional and reliable applications.
Making the Case for Agile BI and DW
A new Forrester Research report on agile BI -- Agile BI: Best Practices for Breaking Through the BI Backlog -- paints a familiar picture.
"As data volumes and information complexity continue to skyrocket, traditional BI tools try hard to keep up with ever increasing and changing demands. But it's an uphill battle -- and BI tools and applications do not always keep up the right level of pace and advancement," the report indicates.
"As a result, the rift between business requirements for on-demand information and real-time decisions, and BI applications and IT support staff's ability to support them continues to grow."
Forrester describes a situation in which the BI and DW status quo is clearly untenable -- in part because it doesn't really exist anymore. BI user constituencies have shifted to more to a self-service approach. The problem, according to Forrester, is that neither in-house IT organizations nor BI and DW vendors have responded appropriately in kind.
"Lines between producers and consumers of information have largely disappeared. Traditional lines of demarcation between information producers -- traditionally IT staff -- and consumers (traditionally business users) are now very unclear, as a significant portion of knowledge workers now fulfill both roles," the report says.
Forrester also points out that "BI tools and applications need to be cognizant and supportive of that trend and provide ways for knowledge workers to fulfill their information requirements via self-service."
In Praise of Self-Serviceability
Speaking of self-service, there's a self-serving aspect to Forrester's report. It was commissioned by a search technology vendor (Endeca Technologies) and -- in a few respects -- draws conclusions that might be said to play to that company's strengths. For example, it cites a requirement for a single "simple" user interface -- i.e., a salient advantage commonly touted by enterprise and BI search players.
That being said, the Forrester report draws from survey data that buttresses its conclusions. For example, 51 percent of respondents say that BI requests tend to accumulate (as a backlog) in their organizations, while 66 percent say BI requests accumulate precisely because their IT organization already has too much on its plate.
For this reason, Forrester points to self-service as an especially compelling agile solution. In a BI context, Forrester says, self-service helps address a problem -- namely, the insufficiency of traditional requirements analysis -- which helped give birth to the agile application development movement in the first place.
"Even when one architects a BI application by the book, according to all best practices, it's often still an unattainable goal to make such a BI application react on a dime to constantly and frequently changing business requirements," the report notes. "The sheer number of BI components … and a tight connection [dependency] among them result in a significant effort just to make even a simple change."
In fact, Forrester says, nearly three-quarters (70 percent) of survey respondents say their requirements change on a monthly, daily, or even hourly basis. What's more, 22 percent of respondents believe their requirements change too frequently for traditional BI applications to "keep up."
Forrester also chides the BI industry for not having responded quickly enough -- or in an agile enough fashion, if you will -- to the changing needs of knowledge workers or information consumers. Canned or largely static reports are so behind the times, according to Forrester. Even Jane C. Knowledge-Worker -- as distinct to her business analyst or power-user colleagues -- wants to incorporate self-service analysis into her information diet.
Forrester survey data indicates that respondents in large ($5 billion or more in annual revenues) shops require free-form analysis 45 percent of the time. This requirement is even more pronounced in midsize enterprise shops (those with annual revenues between $2.5 and $5 billion), where respondents say that -- ideally -- they'd consume free-form analysis 51 percent of the time.
"[M]ore often than not, unstructured, free-form exploration and analysis is needed to answer an in-the-moment business question," the report concludes. "Indeed, the survey respondents indicated that only on average, 51 percent of their BI requirements can be fulfilled by a canned, structured production report, while on average, 49 percent of the cases require free-form exploration and analysis."
Self-Service Management
Forrester's self-service recommendation applies to both BI consumers and IT enablers. Just as BI users can benefit from empowerment of self-service, DM pros can benefit from tools that are -- in effect -- likewise self-serviceable. For example, the Forrester report notes, 54 percent of respondents cite the complexity of application change -- specifically, the challenges associated with rebuilding or recreating data integration (DI) mappings, regenerating dashboard views, re-architecting portals, and other issues -- as a major impediment to executing on feature or function requests from BI consumers.
Moreover, fully half of respondents (50 percent) say that they just don't have enough IT resources to respond to change requests, given the complexity of the DM status quo.
Finally, almost one-third (32 percent) of respondents cited inflexible or intractable data models as substantive impediments to change.
The upshot, according to Forrester, is that BI and DW vendors must incorporate self-service-like manageability features into their offerings, with the goal of making things like DI connectivity more tolerant to (or insulated from) change.
"Traditional application … data modeling and data warehouse technologies, while quite powerful, can seldom react in time to support frequent and ever-changing business requirements," the report concludes. "As a result, even the most user-friendly and intuitive BI application may not be able to answer your question, since the application and/or data model may not be changed and populated in time to support the new type of analysis."