The State of Agile BI, Part 1: Agile BI at a Tipping Point
How can we deliver BI solutions to our users faster without compromising architecture and quality?
By Tom Hammergren, CEO, Balanced Insight
Agile BI has been one of the hottest topics in business intelligence circles for the last several years. This three-part series highlights the state of Agile BI today -- why it’s more valuable than ever for developing highly usable BI apps, optimizing information delivery, and creating a stronger, BI-driven, decision-making culture.
There can be no doubt that the promise of business intelligence (BI) offers enormous potential value to nearly every type of business today. Indeed, given the explosion of data volumes and the proliferation of digital technologies and platforms for receiving information, businesses need BI more than ever.
Why, then, do so many of us feel that BI is in disarray? It’s hard to believe that Gartner studies continue to show that 70 to 80 percent of BI projects fail to deliver the BI capabilities users want. Yet, no one will give up on BI: CIOs have pegged BI as the top technical priority through at least 2017, according to Gartner.
Part of the reason is the steady stream of powerful new tools and technologies that are launched almost daily. Many of these aim to remove obstacles to the basic and eternal task of BI -- getting the right information in the hands (and devices) of the right users as quickly as possible. The huge obstacles presented by a range of outdated practices -- in everything from technology architecture and data management issues to sub-par resource allocation and weak business-IT engagement -- remain an enormous challenge. Few companies seem capable of overcoming these barriers.
We are at an inflection point -- still facing a core “meaning of life” question for BI: how do we deliver solutions to our users faster without compromising architecture and quality?
Clearly, it’s time for the BI community to change -- and I don’t mean to simply rush out and buy the silver bullet technology du jour. The vast majority of companies and BI teams already have more than enough technology firepower for effective BI. Rather, it’s time for many BI leaders and teams to rethink their approach to information delivery and programmatically change their BI development process. By merely doing things better, BI teams can deliver much more to users -- more robust capabilities, easier-to-use apps, more confidence in data and, more important, higher business value -- in much less time than it takes to release a BI app today. Many firms will be able to do it without adding costs.
How? Agile is the answer. Yes, enterprises need agile development methodology specifically, but they also need a more generally flexible and agile way of development and greater analytical and operational agility, which is what so many businesses need from BI. This multi-dimensional view of agile BI is the way forward for organizations looking for breakthrough gains in both BI capabilities and the productivity of BI development teams.
Agile BI is best viewed as a concept, not as a specific method. All too often we turn anything agile into narrow conceptions of Scrum or Kanban or Extreme, when we should focus on the bigger picture of finding the most cost-effective release schedule to drive the greatest value out of our business intelligence solutions.
Why Agile Fits BI
Let’s start with the agile methodology and its appropriateness to the BI realm.
Some analysts question whether agile is a good fit. The answer is both yes and no. When you evaluate agile in the application development context, the number of technologies and specialties is significantly fewer than in the world of business intelligence and data warehouse (where often you have 6-10 and sometimes as many as 20 technologies requiring work). I think you must look at agile in the context of scope and accelerators. This is a different perspective than agile in the application development sense. You have to acknowledge that all the technologies and skill requirements force a certain waterfall style cadence to your project; therefore, you can deliver “working features” more frequently by managing scope.
At the same time, however, agile does not mean eliminating all documentation and architecture. Failure often occurs when development teams think these two steps are the reason development goes too slowly. There must be clear understanding of the essence of what is being built and a link to the ultimate vision of what BI should be. That's why the mindset of people working within BI is so important. The bottom line: if you don’t have the right people with the right mindset, agile will never work.
In my experience, however, the agile methodology -- especially when compared to traditional waterfall development -- is uniquely suited to avoiding many of the common pitfalls that hinder development projects and ultimately prevent the realization of the full BI value proposition. Those pitfalls include:
The project mindset of IT, which views information delivery not as a continuum and constant need but as one-off deliverables
Lack of collaboration between business and IT in developing solutions that the business needs and wants
Impossibly broad scope or excessively narrow scope
Lack of clear ownership of the business vision so that the project also fits in the bigger picture, not just a technical vision
A complex mix of technology, with many tools and applications requiring specialists
Narrow definitions of information ownership rather than enterprise stewardship
Insufficient resources to stay ahead of mounting BI project backlogs and deliver faster
The pressure to constantly reduce development costs
Low end-user adoption rates for delivered solutions
The inability to define the end of projects or know when you’re done
Overreliance on virtual teams or an overly virtualized team whose members aren’t organized or incented to collaborate and provide timely feedback
Our experience tells us that projects fail when any of these things happen.
Thanks to its emphasis on iterative development, faster delivery of incremental solutions, and automation and reuse of core components, agile helps organizations navigate these challenges.
Full disclosure: if you haven’t guessed already, I am firmly in the pro-agile camp. In fact, I would argue that many of these longstanding issues in BI are the direct result of sequential waterfall development. That’s why I’ve come to believe that agile has innate advantages in the BI space.
Consider how BI development is much different than standard application development. The ability to be agile is driven by scope, quality of deliverables, time, and money, and in this context automation can overcome various skills gaps. BI is a technical challenge where delivery of a solution might require six to 10 different technologies to come together; this is a significantly different challenge than application development teams face. Therefore, managing scope allows for the ultimate solution to be crafted over time, and the ability to manage scope to deliver value quicker. Acceleration and automation technology should be incorporated into your process to speed the delivery of various technical components and cover skills gaps.
The underlying factor here is the “productivity imperative” faced by development teams. They are pressured to deliver better apps and information faster to business users who need more sophisticated tools to gain visibility into markets and operations that change constantly. In a high-velocity business landscape, agile is simply more likely to succeed, if only because its nature reflects the reality of constant change and it enables the delivery of more solutions faster.
Benefits
Now we recognize that many IT organizations and leaders are skeptical of -- or even hostile to -- agile. I have been accused of drinking the Kool-Aid. However, I believe that the business case is so clear and compelling that even the most “agile-phobics” stand to gain from rethinking their position and adopting some agile practices, especially when it comes to BI.
There are several elements to the value proposition, with resulting benefits for both IT and the business. The tangible benefits include increased developer productivity and efficiency in the form of shorter delivery life cycles and reduced project costs. Those gains are practically built into the agile approach, right from project initiation. Poor, misunderstood, or misinterpreted requirements are the original sin of most failed projects; thus, agile’s forcing of consensus on a discrete set of features and deliverables reduces the risk of costly rework and user frustration.
Similarly, once developers and users agree that a specific feature or functionality is working well, that component can be integrated and reused and/or enhanced in all future releases. Helping to reduce the all-too-frequent “back to the drawing board” moments is the means by which agile helps boost productivity, efficiency, and cost-effectiveness. Thinking in this mindset can easily reduce project costs by 40 to 75 percent alone.
Looking at the business side of the equation, the most compelling value from agile comes from increased information access, which is value that can build and compound iteration over iteration. In one customer situation, we found the first project alone provided a tenfold return on investment and each following project incrementally increased that return. If silver bullets existed, such engagement would be the most likely candidate as it pays off on multiple dimensions:
- Increases trust of information and can even de-politicize data management issues
- Gives users the reporting, query, and analysis tools with self-service capabilities they need to keep up with higher-velocity business analytical and decision-making tempos
Confirms that IT is a partner that will take user and business requests and integrate them into a usable solution in short order
The bottom line is that the business case for agile BI is attractive and achievable for all types of companies, not just those that buy into a certain manifesto. That brings us back to the question if agile is right for your organization. The answer is almost certainly yes if you are interested in dialing up the value BI can deliver, and -- this is a big and -- your organization is willing to addresses the challenges associated with change.
To put it another way, are the natural challenges and potential friction that come with a change in development approaches justified by the potential value that agile can deliver? That specific issue is the agile “tipping point” that the industry has reached. I believe the answer is yes -- the agile value proposition is worth the effort because it’s the best approach for solving much of what has ailed BI in the past.
Part 2 of this series focuses on the "how" -- which steps organizations can and should take to start developing in a more agile fashion and promoting analytical and operational agility as right and worthwhile outcomes for BI programs.
Core enabling concepts for BI success:
- Business involvement: The best BI apps have a real owner with a real business vision (not just a technical vision) for what is being built and why
- Clear sight of the bigger picture: The way to value is incremental, with each iteration growing from the foundation and leading to the creation of larger solutions
- The right scope: Most projects needed a narrower and more clearly defined focus, but not so small that it doesn’t add value
- Clear definitions of "done": Related to the right scope, the business must understand and sign off on what constitutes a completed project
- Non-virtual teams: Because BI is so important to the business and requires deep and nuanced understanding of what users want, it must be delivered by a permanent set of core team members, not an ad-hoc or virtual group
Tom Hammergren is a BI innovator and speaker and the author of Data Warehousing for Dummies, Data Warehousing: Building the Corporate Knowledge Base; and Data Warehousing on the Internet: Accessing the Corporate Knowledge Base. He was a member of the initial Cognos BI team that produced PowerPlay and Impromptu, and a member of the Sybase team that produced the Warehouse Studio product family. Tom is the founder of Balanced Insight and has led major BI transformations and initiatives for companies such as Procter & Gamble, Cinergy (now part of Duke Energy), Quantum Chemical, and FirstEnergy. Tom can be reached at [email protected].