Q&A: Transforming Your Organization into an Agile Enterprise
Agile is a lean philosophy that's useful for more than just development. How can you maximize success when moving to an agile enterprise? PayPal's Kyle Forbes shares his thoughts and experience.
- By James E. Powell
- August 26, 2014
[Editor's note: In his keynote presentation at the TDWI World Conference in San Diego (September 21-26, 2014), PayPal's Kyle Forbes describes his experiences building agile data organizations from scratch and transforming existing organizations, the lessons learned, and simple, tested rules to follow for companies craving more value from their data. BI This Week caught up with Kyle to understand how he came to focus on agile, his suggestions for building an effective agile data organization, and how your organization can maximize success in your own transformation.]
BI This Week: Let's set the stage for our readers. Tell us a about your background and what you do now.
Kyle Forbes: At PayPal, I lead a product owner group focused on technology capabilities that accelerate analytics and data development. I also serve as an organizational and portfolio strategist in the data space, helping guide department methodologies and how we think about our long-term, strategic focuses.
Originally my education was in forestry and wildlife biology, which led me into a graduate degree using geographic information systems and remote sensing to build environmental predictive analytics models. I discovered my passion for engineering, so I ended up focusing more on developing technology for problems than on science.
I spent ten years in the online computer games industry as a product developer, engineering manager, and eventually leader of a central, shared tools and technology development group. It was during that time that I developed a fascination for why some product companies were so successful and others weren't.
How and why did you end up focusing on agile, organizational process, and organizational strategy?
During my time in the computer games industry, I started observing that there was a distinct difference in philosophy between successful product companies and unsuccessful ones. When your job depends on the success of the next product you launch, there's a lot of incentive to find better ways to be successful.
Agile took root very early in the games industry because of the need to constantly innovate with new ideas. Many teams around me were adopting the process, so I started observing how people, day-to-day, came together to develop their products, primarily so I could learn better ways to lead my own teams.
I discovered that some organizations struggled to produce any results and others seemed to glide effortlessly to success. Ever since, I've been convinced that regardless of how smart people are, a strong organizational methodology is the foundation needed to consistently succeed. A leader's primary focus should be on building an organization that follows a methodology for achieving its goals.
It's often said that agile works great for software development but that the techniques don't apply to data development, analytics, and warehousing. With your combined software and data background, what are your thoughts?
Agile is a lean philosophy. It is not a technique for developing software. Agile is a philosophy for how to be efficient at addressing a need, problem, or value opportunity. The need to be effective problem solvers, learning as we go and adapting to the unknowns we inevitably encounter, is universal. Although data development and software development have different technical challenges, the mindset of understanding a need and iteratively trying small, incremental solutions against that need is the same for both.
When working on data or analytics projects, there are typically two reasons teams have trouble with agile. The first is that they've built organizational processes and tools in a way that makes it difficult to simply try something. Everything has been designed to function reliably, and everything is viewed as a production change. There isn't a good core of tools and processes focused on the test-and-learn phases of information development -- everything seems to be geared toward well-defined implementations.
The second reason data teams have trouble with agile is that they have been trained to "implement" a request rather than on how to understand the problem at hand and orient their activities to incremental test-and-learn approaches to solving it. Therefore, we often find ourselves prematurely in conversations about the best way to design an ETL chain before we've ever loaded a single byte of data into a database to see if that helps to address the business question.
Don't ever assume an implementation yields the solution. It will certainly yield what was requested, but it's hard to know if what was requested actually solves a problem. Our focus should be on whatever way we can get information to the business to see if that addresses the need and, afterward, on the optimal way to harden that process to ensure it is durable and that is has the highest optimization level for its cost.
You say that building an effective agile data organization leads to better business value and a more aligned technology strategy. How so?
There's a lot of talk these days about customer-driven innovation. This is really just another way of saying "go see the problem for yourself so that you know what you need to solve." If you don't understand the problem, how can you possibly create a solution for it? If the team working on the solution doesn't understand the problem, the solution is going to be approximate at best.
By teaching an organization to work in teams to tackle problems -- which is what agile does -- we create the foundation for an organization that changes from "tell us what you want and we'll build it" to "tell us about the problem and we'll innovate to solve it and evaluate our success along the way." In doing so, everyone progressively becomes more closely aligned with the value the company is trying to achieve, not just the implementation of some request that may (or may not) yield the intended value.
Inherently, strategy is about seeing the bigger and longer-term picture of the organization's top issues. Once an organization learns how to align with problems and tackle them as teams, aligning them with higher-level, longer-term, strategic goals becomes infinitely easier, and they will use their intelligence and creativity to efficiently identify ways to achieve those goals rather than blindly implementing what they were asked to implement based on an assumption about the solution.
We hear a lot about failed transformations. What are the top things an organization should pay attention to in order to maximize success?
For the teams I was building, it was fairly easy to create them as agile teams, but it is significantly more challenging to transform an organization of 400 people, or more, to agile. The mechanics of the transformation are easy: put people in teams, define the roles, and establish ceremonies. That teaches the delivery organization how to operate in a basic agile mode.
Unfortunately, there's very little point creating an agile delivery organization if organizational and portfolio management decisions are not routinely made using that methodology. Take a simple example: hiring a new manager and not teaching them about how their agile organization works. Suddenly, that manager starts managing their direct reports in the way they are used to, not in a way that supports building agile teams.
We often see companies in which some groups have managed to "go agile," but all of the mentality and actions surrounding the agile teams (particularly at the senior leadership level where critical decisions are made) compete with or slowly erode the team's methodology. For an organization to be truly agile, everyone has to operate in that mindset and mechanics, not just developers. All of the practices, such as budgeting, portfolio planning, and project evaluation, must be changed. It sounds like a lot of work and it is, but it's worth it.
Doesn't process tend to get in the way when the organization needs to move fast? How can we avoid throwing it out when the heat is on?
That depends what your processes were designed to do. We shouldn't forget that the purpose of process is to create a repeatable methodology for any activities considered critical to organizational success. In the case of agile, the methodology is designed to make organizations extremely adaptable and focused on getting things done efficiently. To throw out this methodology when the "heat is on" is the mistake. It is exactly at this time that you want to follow the principles of agile, keep the team focused, aligned, inspecting, and adapting.
I have personally experienced this situation, where the heat was on and everyone wanted to forego the process and just throw everyone at the problem. We know this results in chaos and poor results. A seasoned agile organization will understand how its methodology allows it to perform well in this situation. Organizations that tend to throw out the process and do whatever it takes to get things done are still learning, and it's a sign that more time and education are needed. Don't ignore opportunities to collectively reflect on the desire to throw out the methodology.
How are things going with the organization you're in now that it's "transformed"? Is it all done and working, and what's next?
Despite all of the things that still don't work as well as we'd like, an impressive amount of positive change has permeated every area of the department. We enjoyed benefits we never expected. We've made it through the early phase of getting our delivery teams into an agile mode. All of the roles and ceremonies are second nature at this point. People understand and embrace operating in this mindset.
In retrospect, however, we mistakenly focused a lot on the agile teams and afterward realized the transformation had to start at the top to ensure an understanding of how the new organization would work and what core practices would need to change for it to function properly. While you're busy getting forty teams into an agile model, other teams are being created in an entirely different model simply because organizational leaders aren't trained in the new approach and how to make decisions that align with it.
We're still untangling a few of those things, but mostly we've moved on to focusing on how to get an organization to function as a cohesive unit through clear, core practices such as agile portfolio management, ensuring teams have well-defined purposes that align with problem-solving (not other methods of grouping people), and routine evaluation of how well the organization and its teams are operating.