Q&A: Agile BI Practices in the Real World
A preview of TDWI/Ceregenics research into the broad trends in agile methods adoption and how to select your first agile BI project.
- By James E. Powell
- September 17, 2013
[Editor's Note: Ralph Hughes, chief solutions architect at Ceregenics and Dave Stodder, director of research for business intelligence at TDWI are speaking at the Thursday keynote address TDWI World Conference in Boston (October 20-25, 2013), We spoke to them to get a preview of their presentation and to learn about what best practices enterprises are employing in their agile BI projects.]
BI This Week: In your keynote to the TDWI World Conference in Boston, we will be hearing about the 2013 update to the joint research survey by TDWI and Ceregenics, Adoption of Agile Methods among Business Intelligence and Data Warehousing Development Teams. We will have to wait for the keynote to hear your actual survey insights. Until then, given the results you've seen come in, what are some broad trends in the adoption of agile methods? What are some of the signs that organizations are successfully sustaining their adoption?
Ralph Hughes: The survey shows that agile methods increase programmer productivity, application quality, and customer satisfaction across all industries and company sizes. My personal experience is that they double or triple BI/DW delivery speeds. Scrum is the dominant method, followed by Kanban; this survey result matches what we're teaching at TDWI.
The results also show that a few projects will give organizations experience and insight, so companies just getting started should network with individuals who've done it before. We are also seeing that new data modeling paradigms such as hyper-normalization and hyper-generalization are allowing organizations to as much as triple delivery speeds, so organizations should evaluate whether to make them a standard part of the agile data warehousing toolkit.
Dave Stodder: One sign of growing adoption goes beyond just the survey results. At the recent BI Executive Summit in San Diego (in August 2013), TDWI featured several case studies of agile method adoption. Even just a year ago, we were fortunate to have one such case-study presentation; at this year's Summit we had five. In the survey results, we are seeing that more organizations are exhibiting greater experience in terms of the length and number of agile projects. Anecdotally, we are seeing growing interest in adhering to the broader objectives agile, which is then often followed by more rigorous use of an agile method such as Scrum or Kanban.
The impact on costs was a concern that you noted in last year's survey results. Do you expect this to continue to be a concern regarding agile adoption, even as organizations report gains in developer productivity, application quality, and customer satisfaction?
Ralph: First, viewed in the aggregate, only some industries and larger companies struggled with project cost. We've added a question to the 2013 survey to get more details on why. Second, there were many respondents who said they experienced improvements in all areas at once, including cost. I see this as a challenge that our community can solve by researching the extra practices that these superstars have adopted. We'll be adding those special practices to the TDWI agile course material as we identify them.
Dave: One insight from our research is that it is difficult to look at the cost issue in isolation. Sometimes personnel costs rise because organizations are using traditional project management accounting of individuals' time spent, which may not properly reflect the way teams work together on agile projects. In addition, looking at costs alone will not reveal the overall business productivity benefits and faster path taken to delivering applications that meet user needs. Thus, the incremental, team development approach make it appear that immediate project costs are higher, but the rise will be balanced out by greater efficiency in reaching objectives and higher satisfaction from users.
Business-IT collaboration is a critical success factor in making business-driven BI work. How does agile adoption contribute to business-IT collaboration? Does your research suggest that agile development teams are getting the collaborative leadership they need?
Ralph: Respondents who said they get sufficient access to business experts outnumber those who do not by only 5 to 4, so clearly this constraint is an important issue. Unfortunately, generic agile methods offer little to address this problem. The consultants at my company have had to infuse agile with a handful of templates for requirements management that provide a more dependable means of interacting with the user community, and that has helped. We've added those templates to the TDWI classes I teach on agile requirements management.
Dave: Without good collaboration -- or worse, where real antagonism exists between business and IT -- BI/DW projects can run up against obstacles or misunderstandings that go unresolved and become worse. Development, testing, and deployment steps are out of sync with changing user requirements. TDWI Research has found that most organizations do acknowledge that collaboration is important. However, they have trouble knowing what steps to take to improve collaboration. Agile methods give organizations a model for business-IT collaboration; agile requires that business users play a serious role in all phases of the development cycle. We are finding that the partnership fostered by agile method implementation can be more easily sustained because there is a template. Agile can improve developer productivity, which is in the business users' interest.
How does agile adoption fit with other key trends in BI and data warehousing, such as self-service BI, demand for more rapid development, and business-driven rather than IT-driven projects?
Ralph: First, agile done right embeds a business partner in the project team, so that alone greatly helps with business-IT alignment. Additionally, we've added some BI/DW specific practices that utilize a new generation of productivity tools, into an "agile BI/DW release value cycle" that extensively puts data visualization ahead of ETL and lets programmers "back fill" the architecture later. This greatly helps make the entire agile BI delivery more user-driven and self-service.
Dave: BI/DW systems are critical to enabling business users to respond to faster decision cycles, to competitive pressures to seize fleeting opportunities, and to unexpected changes in customer demand, supply chains, and more. Yet, if development of these systems is slow and inflexible, organizations have a problem. Agile method implementation is one part of solving this problem. Giving users self-service BI, data discovery, and data integration tools that allow them to access, analyze, profile, cleanse, transform, and share information without having to wait for IT developers to produce applications is also important. The more users can dig into data and create visualizations on their own, the more productive they can be. Self-service tools don't solve all problems, but they can be an essential component of improving overall agility.
What types of projects are best to start with for implementing agile methods? Which projects offer the most visible bang for the buck from a business user's perspective?
Ralph: Agile methods shine any time a company needs to get more than a few people to collaborate on a goal that entails a significant number of unknowns. That's a huge portion of what we do every day in BI/DW. More specifically, I like to start my client's agile EDW transitions with projects that are pressing, important, and have high visibility. Then, faster delivery will mean something to someone important. There may not yet be a crisis in this project, but without a change in method, it could be a disaster. You could save it.
Dave: I would agree with Ralph that there's no reason not to aim agile method implementation at meaningful projects that are currently in trouble. Look for projects where there is an urgency to improve the speed, productivity, and quality of development. In addition, because it is so important to create small, cross-functional development teams to implement agile methods, look for projects where it will not be too difficult to physically bring together, in one location, stakeholders from all the relevant business and IT functions. Make sure to explain what is going to happen and to set expectations clearly about how iterative development cycles will work and what team members are expected to contribute.