TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

Page 2 of 2

An Imperative to Build, Not Buy, Agile BI

• August 4, 2010

At this month's TDWI World Conference in San Diego, agile business intelligence (BI) will be front and center. [Editor's note: "Creating an Agile BI Environment -- Delivering Data at the Speed of Thought" is the conference theme.] Keynote speaker Wayne Eckerson, director of education and research for TDWI, will discuss the benefits of an agile BI environment.

TDWI San Diego also has several agile-oriented courses on deck, including "An Agile Method for Data Warehousing," "Agile Data Warehousing Survival Skills," and "Architecture and Technologies for Agile OLAP."

Recent studies suggest that BI uptake has basically flat-lined. () Agile proponents cite a litany of other (familiar) complaints, too: IT departments are struggling to do more with less. Business users want greater flexibility. They also want adaptability, responsiveness, and alacrity.

A new report from consultancy Forrester Research succinctly frames the issue. "BI tools and applications do not always keep up the right level of pace and advancement," concludes Forrester analyst Boris Evelson. "[T]he rift between business requirements for on-demand information and real-time decisions and the ability of BI applications and IT staff to support those requirements continues to widen."

The Forrester report -- Agile BI: Is It Time to Make the Move? -- is the latest entrant in a still-cresting agile wave. It's an especially interesting entrant, however, in that it seeks to explore both the impetus and the contextual feasibility of going agile. Evelson notes that Forrester sought to "test the assertion that a different type of BI architecture and application development and deployment may be needed to support the continuously growing hunger for information while relieving reliance on IT staff."

To that end, Evelson and Forrester conducted surveys with more than 200 business and IT executives. Not surprisingly, most respondents confirmed that they experience "BI decision-making shortfalls," and -- although Forrester identified several reasons why this might be the case -- it also concluded that the adoption of agile concepts or methods could address (or at least limit the scope of) these shortfalls. "Agile BI technology … is the key to enable BI self-service for knowledge workers, address multiple shortcomings of traditional BI applications, and significantly reduce BI backlog," Evelson writes.

Why agile? More to the point -- why agile now? Think of it as a practical manifestation of what some like to call the "Feiler Faster Thesis." This is a claim that the pace of change in a society is determined by (or is in some sense the product of) the pace of information production.

The faster that information is produced, the faster it's consumed.

Ditto for information consumption in the enterprise. As TDWI's Eckerson and others note, it isn't just that the pace of information consumption is skyrocketing, it's that one practical effect of the requirement for (faster, more timely, more accurate) information is to likewise complicate the development of BI or data warehousing projects. One upshot of this is that -- although the period between a project's conception and a project's delivery invariably produces change requests -- these days, BI projects are veritable moving targets.

For one thing, note Evelson and Forrester, a full two-thirds (67 percent) of BI requirements change on either a monthly or a daily basis.

Drilling down even further, one-fifth of participants (one-third of whom are at the IT director or above level) said that their BI business requirements change on a daily basis.

Far from being a responsive or proactive discipline, BI practices have become increasingly reactive: almost three-quarters (71 percent) of survey respondents said that they have to ask analysts to create custom reports; additionally, more than one-third of custom reports require custom cubes or data marts. More than three-quarters of respondents said that IT can take days or even months to process requests.

Agile Questions

Forrester identified a number of questions that CIOs should ask themselves before exploring agile BI or DW approaches.

There's a clear sense in which most of these questions -- e.g. "Do you feel frustrated that you can't react in time to constantly changing business requirements?"; "Are you tired of hearing that every new BI business requirement … results in changes to multiple programs and other components?"; "Do you hear about your business analysts, developers, and DBAs being locked in meetings for months trying to come up with enterprise data models?"; "Do you have a BI request backlog that never seems to go away?" -- are self-answering.

Based on Forrester's methodology, almost all organizations should investigate agile concepts and methods. This isn't as heretical as it might sound, however. After all, Forrester emphasizes, "agile" isn't a rip-and-replace technology proposition: one doesn't buy "packaged" agile BI. (There is, in fact, a distinct sense in which application "packaging" is inimical to agility.)

"Agile BI is not a replacement for the existing BI infrastructures that most organizations have in place today," writes Evelson. He instead advocates an agile philosophy. "[Agile] is a methodology and technology that helps IT organizations improve their ability to handle complex data, speed the delivery of analytic applications, and put more intuitive tools in the hands of an ever-increasing audience that needs better decision-making tools."

In other words, one doesn't buy agile -- one does (one in effect becomes) agile.

"An Agile BI environment can solve one of IT's longtime bugaboos -- dealing with the problems of constant change: continually fielding requests for new views, reports, and applications and wrestling with conforming new data sources, data structures, and content," Evelson concludes.