Agile Inevitability
A new report from TDWI Research looks at what's driving the need for BI agility. It also assesses how "agility" is commonly conceived -- and best achieved -- in a BI context.
- By Stephen Swoyer
- January 22, 2013
A new report from TDWI Research looks at what's driving the need for BI agility. It also assesses how "agility" is commonly conceived -- and best achieved -- in a BI context.
"Many organizations are under pressure to adjust strategies and tactics amid fast-changing markets, shifting customer preferences, new regulations, and economic uncertainty," writes David Stoddard, director of TDWI Research, in TDWI's Achieving Greater Agility with Business Intelligence best practices report. The problem, Stoddard explains, is that few enterprises are substantively agile: one in six respondents to a recent TDWI survey rated their ability to quickly adjust to changing conditions -- i.e., their "agility" -- as "poor."
If "agile" means an ability to rapidly (or easily) adjust to changing circumstances, just one in 10 respondents described their organizations as "agile." This must change, argues Stoddard: "To improve, organizations need to uncover where they have blind spots or are missing signals amid the noise; where choke points restrict the flow of integrated information to decision makers; and where current processes, rules, and practices need to be optimized to fit new circumstances."
For the vast majority of organizations, "changing circumstances" means one of two things: economic or global instability (cited as a chief concern by 72 percent of respondents) or increased competition (likewise cited by 72 percent of respondents). In addition, more than half (59 percent) of respondents cited regulatory change as a key disruptive challenge, and a full fifth (21 percent) described it as "very disruptive."
Organizations also see financial transparency as an important driver for, and challenge to, agility: three-fifths (60 percent) cite it as a disruptive factor; for others -- e.g., for the nearly 17 percent of respondents who described their agility as "poor" -- financial transparency was cited as the top concern.
Is Agile Inevitable?
The TDWI report assesses agility in the context of three established trends.
The first is the availability and adoption of self-service or BI discovery tools: For several reasons, not least of which is pushback from an increasingly frustrated user base, BI vendors and IT organizations are giving users more flexibility to access, manipulate, and analyze data.
The second trend is an ongoing revamping of the architectures that power BI and analytic programs: the traditional data warehouse-driven model is giving way -- and, in some cases, yielding altogether -- to new architectural approaches. In this scheme, technologies such as data virtualization (DV) and disruptive trends such as big data are transforming not only how users access data but how much or what kinds of data they're able to access.
The third trend is what most people tend to think of when it comes to "agile," at least in an IT context: agile software development or project management concepts and methods.
The survey held some surprises. For example, agile BI is established: slightly more than half (51 percent) of respondents to a separate TDWI survey indicated that they have at least one year's worth of experience with agile BI. (Intriguingly, a minority -- 11 percent -- said that they have more than five years of agile experience.) Another 23 percent of respondents said they've been using an agile approach for less than a year, and a quarter aren't using agile at all.
Even though agile BI is established, it's still immature: by far the largest percentage of survey respondents -- 22 percent -- said that they've implemented two or fewer agile projects. On the other hand, a sizable minority of respondents -- 20 percent -- are running (or have run) five or more agile projects.
Although the greatest number of agile BI adopters use some form of the Scrum methodology, there's plenty of diversity: 21 percent of survey respondents (next-most after Scrum) cited "too many influences to name" when asked to describe their primary agile method. In BI and DW, perhaps more than in other contexts, agile adopters tend to employ hybrid or modified methods.
"The specific challenges of BI/DW ... make it important for organizations to find scrum masters who have experience not only with the agile methods but also with BI/DW development. Generic agile approaches can put too much pressure on teams to promote code into release pools after every sprint," Stoddard writes. "Organizations experienced with BI/DW agile development often find that they need some pipelining of work functions rather than strict scrum cycles so that team members have full sprints to address specific requirements."
This jibes with the experience of Michael Whitehead, CEO of agile DW specialist WhereScape Inc., who maintains that agile's goal is agility -- or nimbleness -- in practice. An agile program that's constrained by (or hidebound to) methodology is not, on these terms, agile, Whitehead pointed out.
"There also can be an over-reliance on the methodologies as opposed to the principles," Whitehead told BI This Week during an post-Q&A meeting at last year's Pacific Northwest BI Summit. "You can have scrums and do scrums and not be agile in any way whatsoever ... [and] you can also be agile without scrums."