Making the Case for Agile BI
There's resistance to the use of agile concepts and methods in BI and DW. The reasons for this resistance, agile proponents argue, are misguided.
- By Stephen Swoyer
- July 24, 2012
At this month's Pacific Northwest BI Summit, industry veterans William McKnight and Michael Whitehead teamed up to make the case for agile business intelligence.
Their pitch met with some skepticism.
McKnight, president of McKnight Consulting Group LLC, offered a crash course in Scrum, an agile software engineering model that -- as McKnight teaches it -- can be used in project management, too. "I am looking to try to get a potentially shippable product going as soon as possible and repeating it with every [iteration of the product]," he said.
Product iterations are called "sprints," McKnight explained. In Scrum, sprints are the basic unit of software engineering; they're intentionally time-boxed, such that they last from one week to one month -- and no longer. "When you commit to a sprint, you actually are supposed to share the scope of that sprint out to a broad community," he explains. "You lay it out there, and you actually have to deliver [what you've promised]."
Not everyone was convinced. One skeptic was industry veteran Jill Dyché, vice president of thought leadership with SAS. "Agile is often the excuse for companies that have never been able to apply rigor in the first place to development," she pointed out. "[Adopters have] never been able to define requirements ... [and so] they use it as a pretext for some kind of requirements-gathering."
As a case in point, Dyché introduced the example of a data management (DM) team in one organization that she said succumbed to pressure from other internal groups to go agile. "The data warehouse team was the only team in the company that wasn't agile, so they decided to be agile," she explained. This was "a spectacular failure," said Dyché, in part because the DW team had neither the experience nor the cultural ability to go agile.
Donald Farmer, Qlikview product advocate with QlikTech Inc., echoed this point.
You don't just decide to "buy" or "do" agile; you have to commit to making (sometimes difficult) changes in order to allow agile to succeed. In practice, Farmer suggests, infusing agile concepts and methods into BI, DW, or any other data-centric domain is like deciding that you want to go sailing in a particularly remote -- but not quite inaccessible -- region of the world.
"It's kind of like sailing on Lake Titicaca," said Farmer, referring to the highest lake in the world, which is situated in the Andes mountains. "Once you're there, the sailing part is easy. Before you can [go sailing], you first have to figure out how you are going to get [your boat up] there."
Farmer's analogy gets at what he describes as another potential pratfall with Scrum adoption. "You [can] say the team succeeds or fails, but how do you define the success of a Scrum [project]?" he asks. The sailing project, for example, entails the hardship of actually transporting a sailboat to Lake Titicaca; sailing the boat on the lake is pre-defined as a successful outcome. In actual Scrum practice, successful outcomes can be harder to define. QlikView, for example, uses Scrum to drive its own internal application development; it defines success in terms of user-focused -- instead of software development-centric -- outcomes.
"It's very easy for a team to shift the goalposts during a course of a Scrum and say 'We succeeded at the end'" when, in fact, what they've produced -- while successfully addressing the requirements of a sprint -- is a failure from a user perspective. What QlikView does, says Farmer, is to say that at "the end of Scrum this is how we will know we've succeeded, [and] that's always delivered in user-oriented terms, it's never [based on] an internal [term]."
Agile Anathema?
One intriguing question concerns agile's prospects in business intelligence (BI), data warehousing (DW), or data management (DM), where -- McKnight concedes -- it still hasn't quite caught on. Is there anything about BI and DW that makes agile an especially tough sell?
Maybe. Agile software development first came to the fore more than a decade ago, when proponents championed it as a pragmatic alternative to other entrenched methodologies, including the still-dominant "waterfall" method.
In traditional application development, agile used to frequently bump heads with another popoular software engineering scheme: the Capability Maturity Model Index, or CMMI. On paper, CMMI looked like agile's antithesis; in practice, this wasn't actually the case. It's probably still true, however, that agile is an especially tough sell in organizations -- such as shops that practice CMMI or Six Sigma -- which emphasize the importance of process, top-down control, and repeatability.
This might explain agile's lagging adoption in data management, which has always emphasized a high degree of planning and control -- particularly when it comes to data warehousing. DM teams have been slow to relinquish that control, experts say.
"Back in the early days ... we were developing complete enterprise data models before loading any data onto the warehouse," says Dyché, who was one of Teradata's first employees in the 1980s. "By the time the model was done, the business had evolved and the users were disaffected." Things are better now, she stresses.
"Instead of a whole portfolio of dashboard screens, maybe we'll give one or two dashboards on the company's financial and topline measures [which is] enough to quench the thirst of some financial analyst," Dyché explains. "So essentially what we're doing now is we're deploying things more quickly and more discretely, but we're keeping [what we develop and deliver] 'warm' ... with our constituents: they're continuously engaged because we're continuously provisioning either new BI or new data."
Is there anything quite like CMMI -- i.e., a methodology or scheme that emphasizes process and repeatability -- on the data management (DM) side of the aisle? Take the BI Competency Center (BICC), for example. It doesn't have to be a part of the problem, but couldn't it function to throw up a few roadblocks in the path of a would-be agile express?
Maybe, says Michael Whitehead, CEO of agile ETL specialist WhereScape Inc.
"The issue with a BICC has to do with the process associated with dealing with that Center and from my perspective, the overhead [associated with this process] doesn't add any value to the stakeholder," Whitehead argues. "Most [BICCs] have a building-consent-style process where if you want something you have to apply to them to get permission [to build it]," he continues, noting that this arrangement seems to be at odds with most agile approaches, including Scrum.
The same goes for quality-first methodologies such as Six Sigma. "[Agile] can possibly live in that [Six Sigma] world, but it's not a natural fit. Agile is designed to get around some of the problems [that Six Sigma proposes to address], but in a fundamentally different way," he says.
McKnight, who practices Scrum in his company's engagements with customers, offers a slightly more pragmatic take. "If you just look at Scrum from the perspective of trying to deliver stuff quickly and correctly, certainly anything that puts guardrails up around that aspect of it will be considered holding it back," he concedes. "In my experience, a well done BICC only helps to make sure things are correct, so I welcome that."
Like McKnight, DataFlux's Dyché -- who's no champion of agile -- believes the coexistence of a methodology such as Scrum and an institution such as a BICC is doable, although she thinks both camps will have to give a little ground. In some contexts, for example, the BICC might agree to loosen its grip, or to look the other way, even as Scrum practitioners agree to relax some of their own parameters. Scrum isn't the Wild West, after all: it places a rigorous emphasis on methodology, which -- when it comes to producing deliverables, for example -- can be unstinting.
"The best BI Competency Centers are units of facilitation as much as they are units of development. If you get the [BICC] blackbelts involved in the process, [a focus on producing deliverables with] zero defects just becomes enculturated in your agile process," she pointed out. Think of it as a spoonful of sugar. "Jerry Seinfeld's wife a couple of years ago wrote this great book on getting your kids to eat their vegetables. Her point was that if you just put the broccoli into the brownie mix, they won't even know it's there," Dyché concluded.
WhereScape's Whitehead echoed this point, stressing that agile itself must be agile -- i.e., not overly focussed on rigor or method. "There also can be an over-reliance on the methodologies as opposed to the principles," he said. "You can have scrums and do scrums and not be agile in any way whatsoever ... [and] you can also be agile without scrums."