Q&A: Agile Estimating of BI Projects
Estimating can be a chore -- and produce inaccurate results. Ralph Hughes introduces us to a new approach to the practice: agile estimating.
Estimating is an art more than a science. Unfortunately, results are often inaccurate, so when a new estimating technique comes along, BI professionals tend to be curious but skeptical. At the TDWI World Conference in Boston (September 16-21, 2012), keynote speaker Ralph Hughes will be presenting The Art of Estimating Data Warehousing Projects. In this Q&A, Ralph explains what makes agile estimating so different.
BI This Week: Is there anything new to the topic or are you just going to step through the standard "list and guess" process we've used for decades?
Ralph Hughes: We'll cover the standard process, because there are ways to do a better job at it, but we'll also explore the art of agile estimation, which is completely different, and, in my opinion, far more successful at forecasting labor requirements.
"Agile" is a word that everyone is using these days, but often it's unclear what they mean. How are you using the word?
My favorite pocket definition of "agile" is an incremental software delivery model by which the business customers feel they are constantly pulling value out of the development team. TDWI and I recently polled the 90,000 members and contacts of the Institute and learned that for 80 percent of those companies that have tried them, agile methods make application delivery faster, better, and less expensive.
What's so different about the agile approach to estimating?
Many things. It's based on evidence rather than speculation. It’s top-down rather than bottom-up. It uses a mechanism for the individual data points that's easy (and even fun) for the developers -- you can scale it to big projects without exhausting the participants or losing them in the fog of details.
Is there a difference between agile estimating in general and agile estimating for DW/BI?
Yes. Though estimating for both types of projects end up using the same process to forecast labor requirements, each takes a distinct path to prepare for that process.
General agile projects decompose a project down to what are called "user stories" -- single-sentence expressions of functional needs authored by the business partners. Because they have a multi-layered data architecture beneath their semantic layers, agile data integration projects have to go one step further, decomposing user stories into what I call "developer stories" -- single-sentence statements that identify where the next intermediate form of the data pertaining to each user story persists within the layers of the warehouse.
General agile project teams build their estimates based upon user stories. Agile DW/BI teams use developer stories.
What's the best thing about the agile approach to estimating?
It does what I call "closing the loop." Agile estimating directly incorporates a team's delivery pace, which is measured with every iteration. As the project progresses, delivery speed is re-measured, and the overall estimate updated, leading to an evidence-based assessment of the remaining labor. If the scope or duration begins to exceed resources, everyone finds out about it early, when there's still time and money enough to fix the situation.
You've been teaching agile data warehousing classes at TDWI for almost four years now. What is it about agile DW/BI that keeps you excited about teaching it for so long?
I've seen agile work magic in so many different types of organizations. Even in the first three months of an agile turnaround program, I've had DW/BI directors tell me, "Productivity is up, quality is up, business alignment is better, customer satisfaction has greatly improved." It's easy to stay interested in a method that has such a positive impact on people's work lives.