The State of Agile BI, Part 3: From Agile to Agility
In the last part of our series, we examine the business-driven context for BI and the link between the agile development and a business's agility imperative.
By Tom Hammergren, CEO, Balanced Insight
Agile BI has been one of the hottest topics in business intelligence circles for the last several years. This three-part series highlights the state of agile BI today -- why it's more valuable than ever for businesses and IT teams looking to develop highly usable BI apps, optimize information delivery and to create a stronger, BI-driven decision-making culture.
In Parts 1 and 2 of this series, we examined how BI is at a tipping point and how changing development processes based on agile can pay real dividends to the business. This article drills down into the bigger strategic picture to show the broader, business-driven context for BI and the specific link between the agile development methodology and the business imperative to achieve agility, both operational and strategic. These imperatives, both related to the increased business velocity that exists across industries, are evidence of just how much companies need better, faster, decision-making capabilities.
In my view, the challenges faced within IT today are synonymous with those faced by manufacturing in the years before they embraced concepts such as component reuse, just-in-time inventory, and "lean." It's time for a transformation of how we view BI and how we "do BI."
Why the urgency? Because the business needs BI more than ever and current information delivery approaches aren't working effectively -- and things are getting worse. Unfortunately, most businesses have regressed in how they house, manage, and provide access to data. After years of consensus that data should be integrated and centralized, the pendulum has moved back to massive siloing. Spreadmarts on steroids have proliferated and organizational units are not sharing data.
The challenges are exacerbated when people trying to address the issue -- which is a process problem -- with technology. The information delivery architecture of the future will contain data warehouses, data marts, business intelligence platforms offering query, reports, OLAP, predictive analytics, complex event processing, and big data. All of these operate on data. Effective information delivery processes must recognize that "one version of the truth" will never be captured in one database or one tool. Instead, IT and BI leaders need to think in terms of a carefully designed environment that integrates with the language of the business. Information should be freely accessible in the language of the business in tools that give users the greatest visibility. That BI vision is lacking in many companies, where BI is still viewed mainly as a noun (a tool or report) rather than a verb (analyze or question).
As a starting point, development processes must have consistency across projects and continuity with the bigger picture. We are all building toward that dynamic environment, one release at a time, no matter which tools or technologies are being used. Users actually don't care about technology; they care about gaining insights. (Facebook provides an analogy: users don't care about the many technologies that Facebook is built on; they care about connecting and sharing with friends.) One of the reasons such a limited population of information workers has access to BI -- Gartner has found that it's only 30 percent of business people, despite very high demand -- is that users are forced to figure out what information comes through what tool. We must begin the process of rapidly engineering and delivering the features users need without losing sight of the vision, and that's where agile comes in.
Given the information delivery chaos I've described, the optimal view of BI today is as a broad-based and dynamic analytical and decision-making environment through which information is accessed and analyzed with the end goal of making decisions to improve the business. It's about accessing information for visibility into current operations and identifying trends. For business, the difference is that BI is not a report to be ordered or a specific piece of software. For IT, it means thinking along the continuum of information delivery, not data and systems integration projects. These are shifts in mindset and culture.
Because of the increase in business velocity and data availability, BI must become more adaptive and nimble. "Big-bang" release cycles and waterfall development are dinosaurs in this world. In a real-time, instantaneous feedback world, can BI teams still think in terms of development cycles lasting months or years? If development cycles take that long, significant business opportunities may have already been lost.
Businesses must get more agile -- ramping up (or down) operations more quickly (geographically, seasonally, or for new product launches) -- to seize market opportunities. Changing consumer demands for greater responsiveness in service and faster delivery of products also plays a role. Shifting regulations also factor in because companies must respond to new reporting requirements (that often involve providing new data sets).
Because agility is a term or concept much discussed in business and management circles, it's worth defining it as clearly as possible. Generally speaking, I think of agility as a set of operational and management attributes so that a company can:
- Go to market faster with new or updated products and services (and move out of markets efficiently, if necessary)
- Seize opportunities and nimbly steer clear of risks and threats
- Deliver responsive service
- Establish scalability and repeatability in key functions and areas so successes can be duplicated across lines of business or in new regions
Underlying these capabilities are a few core enablers. The first is the ability to see clearly what's happening within individual parts or across the whole of the business in real time or something close to it. Done right, BI is critical to gaining these insights so you know where and how to move quickly and efficiently, deploying resources or adjusting operations. This "quick strike" operational capability is the second underlying enabler of agility. Organizations that have strong information delivery practices and capabilities have a distinct advantage in the decision making layer; if they are confident in their data, they can move to decisions faster -- without rerunning the numbers endlessly or duplicating (and potentially paralyzing) analytical cycles. In other words, speed and agility are not just operational attributes but managerial ones as well.
Professor Donald Sull of the London Business School has provided a formal and broad definition that links these dimensions. He sees agility as incorporating three elements:
- Strategic: the ability to "probe for opportunities and mitigate risk"
- Portfolio: allocating resources to high-value opportunities to ensure the company is in the right businesses
- Operational: the capacity to "exploit both revenue-enhancing and cost-cutting opportunities ... more quickly, effectively, and consistently than rivals"
Sull also notes that agility enables companies "to succeed, come what may." It is the "come what may" -- the sense that agility helps all types of companies compete in a fast-changing world -- that makes it particularly attractive today.
Here again, it's easy to see where BI fits. When executives, managers, and analysts have high-quality and highly usable tools and reliable data, they can probe for the best opportunities by asking questions of critical business information and prioritize those opportunities. Consider the ability of sales teams to spot trends and patterns in its cross-selling and upselling performance and make the necessary pricing or behavioral changes to drive toward targeted results. Consider how insurance companies can identify the policyholders most likely to lapse and make promotional offers or premium discounts in advance of renewal dates to reduce churn.
These straightforward examples of business agility should be standard operating procedures at many companies. Certainly, most companies have more than enough technology and data to cultivate such capabilities, but too often our information delivery and development approaches get in the way. For example, the most powerful analytical and BI apps aggregate or cross-reference data from multiple systems for broad cross-functional views that allow companies to see both broad trends and interesting anomalies in data about operations and customers. From a development perspective, that can be a great challenge -- unless you use the right methodology, which brings us back to agile development.
Agile as Enabler of Agility (and Better BI)
The central and unique technical challenge in delivering great BI apps is that most solutions require six to 10 different technologies to come together, though the value of data from these systems increases exponentially when they can be intermingled. Agile is uniquely suited to BI because it focuses on delivering incremental functionality to break down the complexity of cross-system integration into solvable chunks. This is just one of several specific and longstanding development challenges agile solves. Excessive backlog, unclear requirements, and "wheel reinvention" development approaches that don't reuse components or leverage automation are others.
Looking beyond the technical realm, however, it's easy to see a direct line from agile methodology -- which is all about delivering new BI tools to the business faster -- to increased business agility. In this sense, the agile development methodology lives up to its name. Operational and organizational agility can be outcomes of agile development. How?
- Agile makes IT more agile. Development teams can move faster and respond more efficiently to shifts in demand, changing requirements, and constant requests from the business. By making development teams more productive, it helps them keep up with reality of increased business velocity.
- From a cultural perspective, agile is necessary to shift the mindset that views BI and analytical capabilities as individual reports or something that the business orders from IT. The end goal is a broader information delivery "ecosystem" where high-quality and critical data is easily accessible to business users who need it and managed and underpinned by effective data governance practices. You don't achieve such an end state overnight or by deploying some new magical, silver bullet technology but by delivering incremental functionality that users need sooner, within the context of a longer-term BI road map.
- More strategically, agile is the most likely -- sometimes it seems the only way -- for IT to generate the increased business value that CIOs have been talking about for years. Beyond keeping the lights on, BI -- or the precision analytical insights effective BI toolsets deliver -- is what the business needs most from IT. Multiple recent studies have confirmed it. BI pros and veteran developers understand it intuitively. Meanwhile, our experience tells us BI must do better. That's why it's time for a change to agile.
Conclusion
I sometimes think that the agile development methodology started off with a bad name. After all, agile is easily confused with the broader sense of business agility. It reminds me of the confusion that sometimes occurs between the IT process of change management and the broader strategic definition of organizational change management.
Still, the similarities in name also highlight a certain conceptual linkage and the specific tangible connections as well. Whatever the semantics here, in terms of delivering what the business needs and helping BI teams generate higher productivity at significantly lower costs, I believe agile is the answer.
The bottom line is that BI is at a tipping point. There is greater demand than ever from the business, but our development approaches have not delivered. With more complexity on the way, it's time for a transformation that will enable BI to overcome these profound challenges.
Tom Hammergren is a BI innovator and speaker and the author of Data Warehousing for Dummies, Data Warehousing: Building the Corporate Knowledge Base; and Data Warehousing on the Internet: Accessing the Corporate Knowledge Base. He was a member of the initial Cognos BI team that produced PowerPlay and Impromptu, and a member of the Sybase team that produced the Warehouse Studio product family. Tom is the founder of Balanced Insight and has led major BI transformations and initiatives for companies such as Procter & Gamble, Cinergy (now part of Duke Energy), Quantum Chemical, and FirstEnergy. Tom can be reached at [email protected].