Reflections on an Agile BI Program
Agile is not a software feature -- it's a process.
by Chris Sorensen
Does this sound familiar: "I told IT what I wanted and all they ever do is hide for 6 months, never to be heard from and then ultimately they do not deliver what I wanted! Their process doesn't work!"
Agile has certainly become the buzzword of the Business Intelligence industry over the past few months. With the craze comes abuse of the term and a slew of misconceptions about what it is, is not and what the benefits are. Suddenly, all vendor products enable Agile! [Editor's note: "Creating an Agile BI Environment -- Delivering Data at the Speed of Thought" is the theme of the 2010 TDWI World Conference in San Diego.]
Agile is not a feature in software. It is a mindset and a process. Like any process, it depends heavily for success on people and the organizational culture in which it is being implemented (see Being Agile versus Doing Agile).
Our business intelligence team has been working under the methodology since 2006 and I believe we still have not hit full stride. What does that mean for your program? Prepare for a long journey. It takes time, dedication, and the desire to improve to achieve the highest state of refinement and that is what we keep working on. In this article I present a collection of facts (in no particular order) that I often touch on when talking to others about our program.
As luck would have it, we were backed into using the methodology via an organizational change. In early 2006 we were moved under the direction of the software development team who had already been living and breathing agile for years. The fortunate part for us was that the culture, success stories, methodology, and expertise already existed. In other words, the conditions for success were in our favor out of the gate.
The roots of agile began in early 2000 with the agile manifesto, but its true roots go back even further in that it borrows from the core principles of lean manufacturing made famous by companies such as Toyota. Just in time, bottlenecks, continuous flow, total quality management, elimination of mudda (waste), and continuous improvements have their roots in the lean movement and these concepts have worked their way into agile.
Development of BI applications is often compared to a restaurant or manufacturing process -- taking raw materials and transforming them into an asset that customers will demand and that can be sold at a profit. BI mirrors these concepts except with data. If we expect our programs to have a long, useful life, they should be proven to generate a positive ROI. The processes build something and the process used to do this can be analyzed and improved using lean techniques.
More than Just Speed
One of the biggest misconceptions about agile is that it is about getting more done faster. This is simply false. It is about delivering the right things of value, with a high degree of quality and in small iterations. The word "more" should be dropped. It is about avoiding waste or "mudda" when creating value. Have you ever delivered something that took a long time to build, only to have it never be used? Ask yourself why that was the case. This is the waste that we seek to avoid.
One of the biggest difficulties is to get the heads of business users and technical teams wrapped around thinking iteratively. Remember that delivering in small bits with communication built into the process is a foreign concept to many. Most people are equipped to deal with big bang and are unsure how to engage with a process that requires constant communication and participation. Others are simply afraid that once you deliver something you will never be seen again, so they ask for everything at requirements-gathering sessions.
Delivering small has other benefits. We can avoid bottlenecks in the process by completing smaller chunks of work and by keeping all points in a process continuously busy as opposed to having too many wait states. This makes it is easier to test and demonstrate. The biggest benefit is that it gets "something" of value into production quicker, versus keeping valuable assets on the shelf in development. If value can be derived, get it into production as soon as your cycles allow. I purposefully refer to the outputs of BI development as assets, and they should be managed as such.
One of the biggest benefits is that when you fail, you fail fast. This is a good thing in that you demonstrate progress periodically and can ask your business users: "Is this what you wanted?" If it is not, you have wasted less development time that you would have under other methodologies (such as waterfall). However there are perception issues with this as failing is generally considered "bad." This is true only if you never learn from failures. If you incorporate continuous improvement ceremonies (such as regular start, stop, and continue sessions), this misperception can be mitigated.
Agile is well suited to data warehouse development because requirements are often difficult to gather for BI applications. This is the nature of BI, coupled with the fact that BI teams are often not properly staffed with dedicated business analysts. Such inherent challenges make adhering to the agile process beneficial. Prototyping, demonstrating, and communicating all help shape requirements over time by showing working models that can be used to illicit feedback.
A word of caution: No process will fully make up for poor requirements gathering.
Architect big and deliver small must be an overarching principle. It is one thing to deliver in small iterations, but you should have some idea of what your end state should look like at the program level. This obviously is the "architect big" part of the principle. The "deliver small" part comes from the agile cycles and data models are part of these cycles (see Architect Big and Deliver Small - Applying Lean Agile principles to BI Program management)
Be Prepared for a Journey
Agile is a process and like any process, it can have resistors. It takes time to hit your stride, so be patient. Agile takes time to implement. Having someone on your team that has been part of a successful agile development process will certainly help. If you do not have any experience, look for a coach who can help guide you through the process up at least get the team trained.
Either way, it is an amazing journey, and it is rewarding to watch a process mature and improve.
Chris Sorensen is the manager of the business intelligence group at WestJet in Calgary, Alberta, Canada where he is responsible for the architecture, planning, and development of the WestJet Enterprise Data Warehouse. You can contact the author at [email protected].