A Spike Takes the Pressure Off Agile
Transitioning to agile development methods can be easier said than done. One way to facilitate the transition is to schedule "spike" stories for planning and discovery.
Many organizations are transitioning, or attempting to transition, to agile methods for data project delivery. This is sometimes easier said than done, as there are many ramifications for your team and your process.
Agile methods require breaking projects down into small steps and assessing progress frequently. Scoping projects at finer levels, with the intention to deliver to production more frequently, requires higher, not lower, organizational maturity. Daily meetings can be daunting when you're always expected to have developments to report.
The thoroughness of waterfall planning is replaced with "shoot first, aim later." In order to embrace agile project management, your stakeholders can't always say "no." They must be ready to find a way to say "yes." Agile requires flexibility and creativity. Team members must still make progress in the face of discomfort.
Agile calls upon cross-functional skills as well. For example, the data integration specialist may need to do some business intelligence work if that is the hyperfocus of the current sprint.
New mindsets, new approaches, new work, and new terminology are the order of the day. To expect to quickly reach the highest maturity with agile is a pipe dream. Teams that are well-established with waterfall methods need to step back and go through forming, norming, and storming before they can expect to perform with agile.
One trick I've found to smooth the transition to agile is to place a general time-bounded "spike" story on the backlog. This takes some pressure off story specification.
When you don't know the details yet, a simple story such as "Spend five hours on metadata planning" can help bring the metadata plan into more focus for the next sprint. It's not always worth the time to detail what those five hours will accomplish, so let five hours ride on planning and then the tasks will be clear for the next sprint because they will follow from what is learned in those five hours.
It's okay to "not know what you don't know" because you will find out during the spike story. I've seen too many sprint planning sessions extended due to work on unnecessary details, which creates frustration with the whole agile process. It's important to keep the agile sessions to their allotted time.
Many stories are formed just before the sprint planning session in response to shortcomings discovered during the sprint demonstration. Because the sprint demonstration is often held very close to the sprint planning, it doesn't allow much time to craft a detailed story. Spikes help.
Spikes also act as a "do not forget" list. The team may know that the project needs metadata planning, but it may not be time yet to spell out the details of the plan into stories. Such stories can end up being very different depending on the metadata architecture chosen, as well as many other early metadata decisions that need to be made.
If the assigned team member is knowledgeable in the subject they are assigned, they will determine where choices are needed, the rest of the team can jump in on the decisions, and everyone will become fully aware of the issue by the next sprint.
Agile approaches are well worth taking for data projects. You will thrive with agile, once you are comfortable with it. The spike is just one small way to adapt agile methodologies to reality, and accelerate adoption.
About the Author
McKnight Consulting Group is led by William McKnight. He serves as strategist, lead enterprise information architect, and program manager for sites worldwide utilizing the disciplines of data warehousing, master data management, business intelligence, and big data. Many of his clients have gone public with their success stories. McKnight has published hundreds of articles and white papers and given hundreds of international keynotes and public seminars. His teams’ implementations from both IT and consultant positions have won awards for best practices. William is a former IT VP of a Fortune 50 company and a former engineer of DB2 at IBM, and holds an MBA. He is author of the book Information Management: Strategies for Gaining a Competitive Advantage with Data.