BI Experts: Relieving Agile BI Choke Points
Organizations must discover why and where their systems are inflexible and how they can apply new methods and technologies to increase agility. These three tips can help you get off on the right foot.
- By David Stodder
- August 14, 2012
Few topics are hotter in business intelligence and data warehousing circles than the implementation of agile development methods. Over the last few years, agile methods have helped organizations accelerate BI and data warehousing development and reduce cost and risk while increasing the quality of deliverables.
Agile differs from traditional, typically “waterfall” development approaches by emphasizing continuous, incremental, and evolutionary growth and improvement. Through use of agile techniques and disciplines, organizations have been able to respond to business change, develop systems faster, and address quality defects sooner -- before they become difficult and expensive to fix.
The agile community within TDWI has been growing; courses on the topic are among the most popular at TDWI events. “Agile BI” is the theme of the TDWI World Conference in Boston (September 16-21). In addition to attending the expert-led sessions on agile at the conference, I am looking forward to sharing the stage on Thursday, September 20 with Ralph Hughes, chief systems architect of Ceregenics and member of the TDWI faculty.
In our keynote address at the World Conference, Accelerating BI/DW Value with Agile Methods: An Inside Look at Trends and Best Practices, we’ll discuss the research survey conducted jointly by TDWI and Ceregenics earlier this year; it examined user organizations’ experiences with agile methods for BI and data warehousing.
Our objective was to gain a real-world understanding of agile experiences, including the challenges organizations face when implementing the methods. As far as I know, this is the first study of its kind, so we are excited to be presenting the findings at the Boston conference. Many thanks to everyone in the TDWI community who participated in the survey!
Discovering Sources of Inflexibility
Whether due to economic instability, disruptive technology change, or the adoption of new strategies aimed at improving profitability, organizations are under pressure to increase the agility of their BI, analytics, and data warehousing systems. Users today are expected to anchor their decisions in solid data analysis. Canned reports that provide old data and little in the way of features for deeper analysis are not enough.
For many organizations, the shift of BI and analytics to the center stage has not been an easy one. BI began its life as a stepchild of online transaction processing (OLTP) systems. Queries and reports would be run in batch at off hours; so as not to interfere with OLTP, analysts would have to find spare machine cycles to perform deeper what-if analysis. Data warehouse modeling and the development of schemas and well-understood data structures have traditionally been a long, painstaking processes.
However, as “competing on analytics” becomes a greater imperative, organizations must find ways of shortening development cycles and providing users with more timely information and analysis. This is what agile methods are intended to address.
Along with evaluating whether to implement agile methods, organizations need to discover why and where their systems are inflexible and how they can apply new methods and technologies to increase agility. Here are three steps that organizations should consider:
Step #1. Root out complexity as a source of inflexibility. It is not always clear why BI and data warehousing systems are rigid and hard to change. Sometimes it is due to a history of an overemphasis on the short term: people needed reports right away, and as more were added, the whole system became sclerotic. “Agile” should not mean more of the same; look for ways to build a development structure that can sustain itself over a long period despite the inevitable sudden requests.
Step #2. Make visibility a necessary ingredient of agility. As systems complexity increases, it becomes harder to discern which applications depend on which data sources. Data systems serving more than one business function or line of business can be hard to revamp if it’s unclear which applications and users will be affected. Organizations should focus on creating visibility into systems and on discovering and documenting data relationships within and across systems. They must also discover who is using reports and analytic processes that access those data sources.
Step #3. Determine whether data integration and ETL processes are thwarting agility. As each new business requirement is addressed, organizations will often amass hundreds -- if not thousands -- of extraction, transformation, and load (ETL) processes until they bog down performance and make it exceedingly complex to make changes. “Orphaned” ETL processes that no longer have an owner or purpose become common. Reducing the morass of data integration and ETL processes should be a key part of improving agility and implementing agile methods.
Think Holistically
In The Fifth Discipline, business management guru Peter Senge wrote that “we tend to focus on snapshots of isolated parts of the system, and wonder why our deepest problems never seem to get solved.” Organizations deploying new methods and technologies aimed at improving agility should be careful not to fall into the “silo” trap and ignore how the parts relate to the whole. The integration and intersections of data systems and what users are doing with them are often the agility choke points; that means that they represent prime opportunities for relieving confusion and realizing higher agility.