Q&A: Agile BI Presents Challenges, Rewards
Faster, more iterative BI projects promising, but take effort.
- By Linda L. Briggs
- May 26, 2010
Agile BI takes a page from agile software development in proposing a faster, more iterative, and more user-involved approach to business intelligence projects. In this interview, BI This Week talked with Steve Dine, founder and president of Datasource Consulting, about the concept of agile BI -- why it's coming to the forefront, how it fits well with cloud computing, and how firms can apply it to what they are doing now with business intelligence.
Dine's company focuses on innovative, lean, and scalable BI implementations and strategic program guidance. He teaches regularly at TDWI conferences and is speaking at TDWI's summer conference on agile BI and cloud computing, and how companies can work to become more agile.
BI This Week: What is your definition of agile BI?
Steve Dine: Agile BI is a project management methodology for software development that can be applied to BI projects and programs. Agile stresses being adaptive rather than predictive, and people-oriented versus process-oriented.
It follows agile software development, a concept that was introduced about 10 years ago and is summed up in the Manifesto for Agile Software Development, a document and philosophy put together by a group of software developers that now has many thousands of supporters. It proposes a new way of developing software through four main principles:
- Individuals and interactions over processes and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan
Agile development employs smaller, more frequent iterations, called "sprints," which typically last between two and four weeks. Requirements are often collected via storyboards, and users are involved throughout development and testing. The project team meets on a daily basis in a process called a "scrum" to promote communication and clear any roadblocks.
One of the keys to successful agile development is providing users with context throughout the sprint, using demos and pilots. This allows users to make adjustments earlier in the process.
You've described the agile process for software development. When we apply that to BI, what are some of the challenges with becoming more agile?
The challenges to agile BI can be both people and IT-related. Agile BI requires strong involvement by the business, as well as an IT infrastructure that can support faster delivery. In some organizations, daily team meetings may be frowned upon culturally, and garnering buy-in may be impossible. Organizations might also have an established project management methodology that all projects must follow. From an IT perspective, the acquisition of server resources, allocation of storage, creation of new databases, and installation of software can often take weeks, if not months. It's difficult to deliver a four-week sprint when developers are waiting three weeks for IT resources.
What are some key ways in which agile BI differs from the way that most BI projects come together today?
One is obviously the implementation methodology. Agile BI stresses much faster iterations, with much greater business involvement. Generally, it incorporates sprints that are anywhere from two to perhaps four weeks long, although some may be longer, along with a daily scrum meeting. The whole team meets for the daily scrums, as well as for the sprints. The scrums basically are used to promote communication and eliminate roadblocks during each sprint.
After each sprint, something new is delivered to the users. During the process, the team also relies heavily on requirements analysis. Rather than sitting and interviewing resources in a more traditional manner, however, requirements analysis is more story-based. We ask the users to come up with their requirements through stories of how they would use BI, what they're focused on, and how it would look. Sometimes they'll use index cards to put their stories up on a wall.
What parts of agile software development don't translate well to agile BI?
One of the challenges we've had with a pure agile approach in BI is the tenet that stresses as little internal documentation as possible. In one project I worked on, the documentation consisted of taking pictures of the whiteboard to create documents. The problem is that with BI, there is so much knowledge that has to transfer within a team, especially if you lose a team member at any point during a sprint and are trying to get someone back up to speed. It becomes challenging when all you have is a set of image files with fuzzy pictures of a whiteboard.
Another area that can be a challenge is automating the test process, which is a big part of agile software development. Although there are tools available to automate system and load tests, it is a bit more problematic to automate unit, integration, and regression testing. The complexities of integrating data arise when you consider automating the testing process.
What are some other differences between how BI is done today and how it would be done with an agile approach?
Today's BI projects often follow a modified waterfall approach -- all the requirements are collected up front, and generally there are very specific sign-offs between phases. The project then goes into a design phase to solidify the design, then into a development phase, then testing, and then rollout.
With agile, all of those steps are integrated into every iteration -- every sprint -- and design is much more fluid. The team is designing as they're developing, and users are providing real-time feedback.
Are users more involved throughout the process?
Much more involved. They have to be, and that's another area that is a challenge. In some organizations, no matter what you do, you'll never get a team to meet every morning for a scrum. It's cultural, and it can't be changed. For one of our clients, that kind of meeting is absolutely impossible. They just don't want to meet every morning.
Overall, that sort of user involvement is much more effective.
On the flip side, a BI implementation can present some additional challenges there, especially in large data environments in which you're integrating a large number of source systems, or you're still working with big, complex data sets. Just cleaning up the data can take a lot of time and throw a sprint entirely off schedule.
You may end up pulling ETL resources to try and populate sample data, but then the team really isn't spending time designing and developing ETL, so that's another area where agile development doesn't translate 100 percent to BI. Part of it is just because of the complexity of the data we're often working with in BI.
Is agile BI something that a project team needs to start right out of the gate, or can it be integrated gradually?
There are different opinions on that. In my opinion, it's something that we try to incorporate into our projects as much as the culture and client will support. We do some full agile BI, but mostly on smaller iterations in which there is an established data warehouse, in which we're helping add new iterations to the warehouse.
It's very difficult, from my perspective, to start out with a full agile approach on an initial data warehouse engagement if you're dealing with a fair amount of data complexity -- with a large number of source systems and so forth.
An agile practitioner might say, "Just select a smaller scope and deliver something." That's great in theory, but the challenge is that a lot of users will say, "Our data doesn't make any sense unless you bring all five systems together." Yes, you can take fewer attributes and fewer entities into your data warehouse but you're going to be challenged to do that in a four-week sprint.
When we talk about BI projects, we often point to the importance of support from top management to move a project forward. Is that a crucial characteristic for agile BI as well?
You almost need that sponsorship even more. Really, it's critically important regardless of whether you're doing a waterfall approach, an agile approach, or something in between. The challenge with agile is that you have to make some extremely difficult decisions more often, which is why you really need that top buy-in.
In agile, you're essentially determining what's going to be the sprint every month, and you have to make very difficult decisions about what is included in that sprint and what's not. As you know, bring 10 businesspeople in a room and you're likely to get five different opinions, so you really need someone that can break the tie. You also tend to have competing interests, so you need someone to settle those.
Remember that you're asking your resources, especially on the business end, to be very, very involved in these projects. That means you have to monitor their bandwidth and availability. It's very easy for management to assign the same amount of work to a resource that might be involved in a waterfall project to an agile project instead, but then they simply can't make scrum meetings, or don't otherwise have time to get as involved as they need to be. You need a really strong sponsor that is willing to step in and say, "Hey, we need to start level-setting our resources."
Let's talk about cloud computing and how that intersects with agile BI.
The cloud can provide nearly immediate access to server resources, storage, database instances and BI software. There are generally three classifications of cloud services:
- Infrastructure as a Service (IaaS)
- Platform as a Service (PaaS)
- Software as a Service (SaaS)
Agile BI project teams can leverage IaaS to allocate new server resources, storage, and software as needed. IaaS providers use virtualization to provide multi-tenant access to hardware resources. Many BI vendors provide server images with their software already configured.
Agile BI teams can leverage PaaS for custom development and alternative database storage, saving time with the configuration of Web servers, application servers, and development platforms.
Finally, there are a growing number of BI vendors providing SaaS alternatives for analytic databases, ETL, and advanced analytics, as well as dashboards, reporting, and analysis.
How can using the cloud help BI programs become more agile?
There is a great deal of continuing confusion and concern about using the cloud for BI. The two most prevalent concerns involve security and performance. Many of the data security concerns can be mitigated with identity access management, encryption, masking, and secure network protocols. However, I suggest that the BI team meet with their internal security resources to understand any compliance issues that might arise with locating their data offsite.
Performance can also be a challenge for large data volumes. Most cloud infrastructures aren't designed for high-speed I/O between servers, and between servers and networked storage. Organizations must have appropriate network bandwidth to the cloud provider. However, even in the short term, BI programs can become more agile by leveraging the cloud for development, upgrades, testing, and proof-of-concepts until they are comfortable moving pieces of the architecture into the cloud.