BI/DW Agility: Go Faster, But Not Too Fast
Organizations implementing agile methods have to ensure that project schedules allow for enough iteration to enable agile teams to uncover and incorporate new requirements into their applications. Only then will the faster pace of development create a faster path to information advantage.
- By David Stodder
- November 27, 2012
Much of the excitement in business intelligence and data warehousing circles today centers on the notion of speed. Technologies and development methods are converging on the potential to dramatically reduce delays in providing users with access to data and the ability to run advanced analytics. From a business perspective, less time spent finding information and surfacing analytical insights means that organizations can respond faster to events in their markets, supply chains, manufacturing processes, and elsewhere. They can manage finances more effectively and stay on top of risks and threats such as fraud or possible regulatory violations.
Of course, the other reason why people get excited about speed is because there hasn't been enough of it in BI and data warehouse development. The common refrain is that BI/DW development takes too long and delivers systems and applications that are so inflexible that making changes to them is even slower. When development goes on too long, business stakeholders either get impatient and covertly fund "shadow" BI and data marts, which can lead to data quality and governance problems, or they lose connection with the project, forget what they wanted, and come up with new requirements that entail a lot of rework to the original designs.
Agile Methods to the Rescue
Problems with slow and inflexible development are driving strong interest in agile methods. Agile methods function with shorter, more incremental development cycles. The cycles produce pieces of software that users can evaluate along the way rather than having to wait until the end of a waterfall development cycle for a final product release or prototype. To quote one of the 12 "points" from the agile manifesto composed in 2001 by some of the top software developers in the industry, "Welcome changing requirements, even late in development; agile processes harness change for the customer's competitive advantage."
However, expectations about the pace of agile development have to be calibrated depending on the type of project. Iteration cycles with the agile "scrum" method for developing BI dashboards can be quick because BI tools are making it easier build applications quickly. Projects for building extract, transform, and load (ETL) or data integration routines will usually take more time and require longer iteration periods because of data architecture complexity.
Using data warehouse development and data quality tools rather than leaning on mostly manual processes could make the pace of ETL and data integration development move more quickly. However, a recent joint research study of agile method adoption conducted by TDWI and Ceregenics found that most organizations are not yet employing advanced tools for data integration and other data warehousing tasks. Technology for data virtualization, which could be a viable alternative for some (though not all) ETL routines, is still not yet in widespread use.
It's important to note that while speedier BI and data warehouse development using agile methods is definitely desirable, it is not the end goal. "Using faster agile methods is not an excuse for doing development wrong," said Michael Whitehead, CEO and co-founder of WhereScape, in an interview for my upcoming TDWI Best Practices Report on BI/DW agility. "You still need to do development properly. Using agile methods is just a better way of getting to where you were planning to go with your data warehouse development in the first place. Agile methods should be a better way of getting there."
Allow Time for Insight
The pursuit of speed should not bypass what can often be a slow and unfolding process of users learning about the data and discovering unexpected insights, which force them to change the requirements for BI and analytic applications. Organizations implementing agile methods have to ensure that project schedules allow for enough iteration to enable agile teams -- ideally composed of IT developers, data modelers, data architects, and business "product owners" who represent a cross-section of the organization's stakeholders -- to uncover and incorporate new requirements into the applications. Only then will the faster pace of development create a faster path to information advantage.