Q&A: The Four Pillars of Agility
Agile is simple but not easy, says chief systems architect Ralph Hughes. He examines the techniques and technologies that can eliminate miscommunications and wasted efforts that often undermine DW/BI development projects.
- By James E. Powell
- September 2, 2014
[Editor's note: Ralph Hughes's Rapid Business Analytics: The Four Pillars for Agility session at the TDWI World Conference in San Diego (September 21-26, 2014) takes a closer look at agile development -- what works and what doesn't.
In this interview, Hughes, chief systems architect at Ceregenics, Inc., previews the four broad categories of agile best practices -- refinements of basic agile methods, complementary software engineering disciplines, new data modeling techniques, and next-generation productivity tools -- that help managers and project leaders understand whether their agile teams will struggle or thrive.]
BI This Week: You're introducing a new, half-day course (The Four Pillars of Agility for the "leadership and management" track at this year's TDWI World conference. What need among DW/BI managers and team leaders are you addressing with this material?
Ralph Hughes: Over the past ten years or so, agile methods have achieved very impressive results on some projects, tripling delivery speeds and driving programming defects to zero. However, I speak with many folks who have had to oversee or lead an agile DW/BI team and have had less than stellar outcomes. Agile teams can fall victim to their own versions of analysis paralysis, scope creep, or flawed outcomes. It seems that there's a key set of best practices that makes all the difference between "doing agile BI" and "doing agile BI right." My colleagues and I wanted to bundle that knowledge into a easy-to-follow presentation, with the hope of making achieving stellar agile data warehousing outcomes a dependably repeatable process.
So your Four Pillars course is a survey of best practices for agile DW/BI leaders?
Yes, but we went one step further and put a bit of an edge on the material. Our experience in leading many agile DW/BI engagements has shown us that failure among these projects comes from one or two sources. If your developers are full-time staff members, the root cause of failure tends to be a lack of discipline in software engineering practices, probably because your team doesn't know that agile versions of these practices exist. If your development team is made up of outside consultants, you have the further risk that there's no incentive for them to build your systems fast and correctly because such performance would only put them out of a job.
With this backdrop, we organized our Four Pillars course to teach those responsible for managing BI development projects the four, basic disciplines that every agile team should be utilizing extensively. We want to empower attendees to insist upon those work patterns in the project rooms they oversee. The conclusion of the course is a BI Buyers' Bill of Rights that these managers can use up-front at the start of a project to set expectations and thereby greatly ensure that their next BI application is built with the all the speed and quality that agile practices can offer.
What are the four pillars that BI project leaders should expect from a high-performance team?
To frame this course, my colleagues and I focused on risk. Given the millions of dollars that are spent on most DW/BI applications, there's always someone whose career either leaps ahead or suffers depending upon the outcome of a project. We asked ourselves what core set of practices should someone in that position look for to know that project is being pursued with the minimum risk? We believe that DW/BI managers and sponsors should look for (a) intelligible project definitions, (b) continual benefits realization, (c) adaptable designs, and (d) quality-driven development practices.
Let's take those one at a time. What do you mean by intelligible project definitions?
Many of the agile methods strongly emphasize putting a business partner on the team and slavishly following their lead as s/he thinks up the features the application should have. Unfortunately, that's tantamount to just chasing the next shiny object. The end result can easily fail to deliver tangible value for the rest of the enterprise. Of course, we don't want to return to the waterfall method's notion of a big design up front, but even agile teams do better if someone provides a high-level vision of what the whole application should look like. Such a vision allows developers to steer straight towards a shared goal.
What is covered by the second pillar, continuous benefits realization?
That's easy. If, as a manager or a sponsor of a big DW/BI project, you don't see working software online until the very end of a project, you are betting everything on one deliverable. There's no reason to take that kind of risk. Agile data warehousing methods have been defined explicitly to enable development teams to deliver a slice of the end product every two or three months. Project leaders should insist upon this delivery pattern. Regular increments of new features generates far more value for the company over the life of a project than a single deliverable after many years of programming.
Moreover, these frequent version releases allow business stakeholders to catch areas where the developers have misunderstood the requirements. They also allow end users to try out the software and tell you whether the new features will really make the business more effective or competitive in the marketplace. In the course, we cover the agile work patterns that allow teams to provide a steady stream of such finished capabilities.
I'm intrigued by the next pillar, adaptable designs. Aren't the designs of large data warehouses necessarily complex? It seems like a contradiction to ask teams to build something that's both complex and easily adaptable.
Let's look beyond just building a data warehouse. What happens if you create a big BI application quickly, load it with 100 billion records, and then your business changes? It costs a fortune to convert all those records to a new design. Even worse, if the warehouse was built using one of the traditional data modeling techniques, it will take months to redesign all underlying database table affected, and months while your company suffers costly paralysis and loses customers to the competition. Anyone overseeing a serious DW/BI development project needs to know that there are two, new data modeling techniques that allow developers to make such changes quickly and economically.
Finally, there's quality-driven application development. This sounds odd. Doesn't testing usually come at the end of a project?
True, most waterfall projects and even many agile teams neglect quality assurance until they draw closer to the end of a project, but planning out testing at the beginning of a project works miracles. It forces your team to think through requirements and designs one more time -- and they always find something important they overlooked or got wrong. Why would anyone want to leave such a crucial discovery till the end of the project? DW/BI managers and sponsors have every reason to demand their teams utilize QA-led techniques that push these problems to the surface from the start.
My colleagues and I have devised a combination of top-down and bottom-up QA practices for DW/BI. We've also figured out how to automate much of it. This course introduces these techniques so attendees can intelligently demand their project teams invest in QA from Day 1 of every project. Working with a team that understands and executes quality throughout the development process is one of the best ways to eliminate the risks of any large DW/BI project.