The State of Agile BI, Part 2: Getting to Agile
How to change development processes to increase business value and speed information delivery.
By Tom Hammergren, CEO, Balanced Insight
Agile BI has been one of the hottest topics in business intelligence circles for the last several years. This three-part series highlights the state of agile BI today -- why it's more valuable than ever for businesses and IT teams looking to develop highly usable BI apps, optimize information delivery, and create a stronger, BI-driven, decision-making culture.
The first article in this series addressed the business case of agile BI -- why it's uniquely suited to addressing many longstanding challenges that have plagued BI development teams and what value it can deliver to the business (including increased operational and analytical agility). This article looks at the "how" of agile BI -- what changes to the development process IT leaders and BI teams need to make.
There is little doubt that business intelligence has plenty of upside opportunity. Recent analyst reports highlight how there is a great deal of demand from the business for analytical insights and increased visibility into the business. The struggles of IT organizations and BI teams to meet the demand and deliver apps that the business loves underscore the need for a change in approach. We see that approach as requiring new thinking (like moving away from project-centric thinking) and new behaviors (such as increased collaboration), which we will highlight. It's about establishing a vision of BI as a dynamic, information-rich ecosystem where analysts and executives live and evaluate and make decisions about the business. The job of BI teams is to deliver that vision one iteration at a time -- and agile is the best way to do the job.
To advance this new vision, several things in an enterprise must happen.
Rethink What BI Is
Perhaps the most fundamental change is a shift in perspective regarding what BI is. Specifically, BI needs to be seen as an ongoing management discipline built on analytical capabilities, not a technology, report, or project. That requires IT to transform its thinking away from the predominant project mindset. The velocity of business cycles and constant rate of change faced by nearly all companies have necessitated this change. The apps BI teams delivered serve to facilitate an analytical process.
BI leaders must adjust their thinking and help their teams see how see information delivery is the continuously operating engine that drives BI. It will take some education, perhaps even evangelism, to do away with the view that BI is a standalone tool, dashboard, or report that is upgraded periodically via major release cycles. In fact, release cycles aren't about a new version number but rather about new features and functionality. It sounds like an obvious point, but by emphasizing it, leaders can drive home the new view of BI. For generations of IT professionals raised in the waterfall development methodology, it is a major adaptive challenge.
Focus on Feature Delivery
One big way to alter the project mindset is to emphasize feature delivery. Agile can help here because of its emphasis on incremental improvements over "big-bang" releases. Such small, meaningful features can include links to new data sources and repositories or adding different "views," such as "by-week" analysis of sales figures, where previously only monthly views were available.
For agile to take hold and deliver value, however, it must become an institutional rallying cry or a cultural expectation that IT will deliver new features and functionality more often with the goal of incremental improvements along the information delivery continuum. In this sense, agile is as much a change in culture and thinking as it is a process change.
Collaborate and Communicate (Especially with the Business)
Thanks to Twitter, Facebook, and other digital media, we now live in a world where instant feedback and immediate, all-the-time interactions are the norm. Business users and IT development teams are an exception to this collaborative environment, however, and many of us know the results (slow development cycles, missed requirements, and low user adoption and satisfaction rates). I often refer to this as the Venus and Mars of BI: one talks in "engineering-ese" and the other in terms of business decision-making. Low end-user adoption rates of new technology and the perception in the business that IT simply can't deliver what users need. After all, how many BI veterans have heard the requirement "give me all the data because that is what I need." In some cases, the translation is actually more along the lines of "You are too slow to translate what I need , so just get out of my way and let me do it with all the data!"
Such collaboration should be built into the way IT does BI. The business-IT communication gap must be bridged by a support model designed for frequent interaction. Developers and business users should get together to discuss what the business needs -- what questions they are seeking to answer, what operational data is hard to access or difficult to verify, how they would like to see information presented. This requires some ability by IT to speak and understand the language of the business -- and to then translate it to specific requirements. When these interactions are frequent and focused on targeted features or upgrades, the shift to collaboration (and away from project-driven thinking) is underway.
The bottom line is that collaboration -- one of the central tenets of agile -- drives faster delivery. The key is to think in terms of a multiple-iteration solution. The first iteration should be to deliver the basics of what user needs. Requirements should be defined in clear terms to understand what "done" means, which leads to test-led development and more assurance that quality is "built in." This may be done via an in-memory database or federated query. In other words, the first step isn't always and likely shouldn't be ETL; exposing data quality issues is a good thing because users typically create quality problems and may very well be better at fixing them.
Embrace Practices that Simplify and Streamline
As companies have continued to throw the latest, flavor-of-the-month technologies at delivery problems, they've widened the existing resource gap by creating new demand for specialized expertise. It's time for IT to embrace automation, accelerators, and model-driven approaches to get them past this resource crunch. Another benefit -- perhaps the most valuable one -- is that these practices also promote reuse and integration.
For instance, tools that can generate working prototypes from a core set of requirements give users a chance to provide feedback instantaneously and avoid those dreaded "back to the drawing board" moments for developers. Similarly, once core components of a solution are completed, they can be reused as building blocks for newer iterations. Such reuse is critical for increasing the productivity of BI teams by eliminating rework, and it improves the consistency of meaning of information across the organization.
The resource crunch can also be detected in that yearning for superstar players who can come in and rescue BI projects gone bad or otherwise move BI dramatically forward. The underlying issue is that there is no need to clone the superstar or duplicate the necessary skills needed to raise the bar of all practitioners. Here again, the reuse of critical components helps, as does ongoing collaboration with the business. These are enabling elements of the agile approach as well as for designing and building better solutions that consistently work together and, therefore, are better for users.
Share Information Freely (Because it's a Resource)
At too many organizations, information is hoarded in narrow silos. These generally align to functional areas -- sales, marketing, operations, finance, etc. -- or product lines and business units. Of course, information is a critical asset for all kinds of business and the true lifeblood for many. Yet, information is too often treated like office supplies.
For BI to realize its fullest potential (and for projects to succeed via agile approaches), the boundaries of project ownership must be expanded toward enterprise stewardship and broader governance approaches. The big idea is that BI is not a tool or technology but rather an environment that business people need to live in, so the information delivery that is the engine of BI must be managed holistically across the full business.
How to Get There: Applying Agile Principles to Your Next BI Project
Even for anti-agile stalwarts, the agile manifesto is accurate in terms of describing and offering potential solutions to address many challenges.
Individuals and interactions over processes and tools: You already have the tools you need; it's more a matter of getting teams working together consistently with same positive experience.
Working software over comprehensive documentation: Radically rethink time cycles and start releasing fewer functionalities faster; get the tools you need to prototype more rapidly; don't ignore architecture but rather think reuse of quality components; the focus on delivery over documentation actually highlights how user-friendly and intuitive BI apps must be today. The documentation should capture and describe the essence of the solution rather than serve as a CYA document full of legalese that basically says "Don't sue me if this thing you are signing off on isn't delivered."
Responding to change over following a plan: The BI reality is that faster is better; this is not big-bang software implementations or a major release cycle; BI requirements change as quickly as business needs to analyze the business; agile respects this reality and gives developers a chance to keep up.
Business people and developers must work together daily throughout the project: Regular interactions and collaborations get organizations past project-driven thinking and establish "sustainable development" tempos. In today's world of collaboration across media, this doesn't actually mean co-locating everyone as much as it means promoting frequent collaboration, responding rapidly when input is needed, and assuring time is properly allocated to drive success.
Welcome changing requirements, even late in development: This is the BI reality and agile processes can harness change for improved outcomes.
Our highest priority is to satisfy the customer through early and continuous delivery
of valuable software" and "Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale: This is the end state to which BI teams should aspire and drive because that's what the business expects.
The Bottom Line: Getting to Value through Agile
As you can see, the value proposition for agile BI isn't a simple matter of developer preferences but rather significant -- even game-changing -- impacts more broadly across the business. Of course, to drive the breakthrough improvements that are possible, agile BI must be supported by effective architectures and clearly defined semantics so the meaning of business terminology is consistent and widely understood.
To recap, the key steps to driving your organization to become more agile include:
- Transform your thinking to see BI not as a technology but as an ongoing management discipline with associated analytical capabilities
- Alter the project-centric mindset in IT to refocus on feature delivery in the context of the overall BI vision, which is fulfilled one iteration at a time
- Bridge the business-IT communication gap with frequent user engagement support models that are enabled via collaboration
- Embrace automation, accelerators, and model-driven approaches to augment current staff and increase reuse and integration
- Drive your BI to an environment that business people need to live in
Some of this work may not be easy, but experience confirms the payoff makes the effort worthwhile.
In the last part of this series, to be published next week, we'll highlight how agile can drive real business agility, placing the use of agile BI in a broader, business-driven strategic context.
Tom Hammergren is a BI innovator and speaker and the author of Data Warehousing for Dummies, Data Warehousing: Building the Corporate Knowledge Base; and Data Warehousing on the Internet: Accessing the Corporate Knowledge Base. He was a member of the initial Cognos BI team that produced PowerPlay and Impromptu, and a member of the Sybase team that produced the Warehouse Studio product family. Tom is the founder of Balanced Insight and has led major BI transformations and initiatives for companies such as Procter & Gamble, Cinergy (now part of Duke Energy), Quantum Chemical, and FirstEnergy. Tom can be reached at [email protected].