Revolutionary Business Intelligence: When Agile is Not Fast Enough
Too many BI teams are shackled by outdated modes of industrial organization
- By Wayne Eckerson
- April 8, 2010
Liberating the Means of Production
Developers of BI unite! It is time that we liberate the means of BI production from our industrial past.
Too many BI teams are shackled by outdated modes of industrial organization. In our quest for efficiency, we’ve created rigid fiefdoms of specialization that have hijacked the development process (and frankly, sucked all the enjoyment out of it as well).
We’ve created an insidious assembly line in which business specialists document user requirements that they throw over the wall to data management specialists who create data models that they throw over the wall to data acquisition specialists who capture and transform data that they throw over the wall to reporting specialists who create reports for end users that they throw over the wall to the support team who helps users understand and troubleshoot reports.
Flattened BI Teams
Contrary to standard beliefs, linear development based on specialization is highly inefficient. “Coordination [between BI groups] was killing us,” says Eric Colson, director of BI at Netflix. Colson inherited an industrialized BI team set up by managers who came from a banking environment. The first thing Colson did when he inherited the job was tear down the walls and cross-train everyone on the BI staff. “Everyone now can handle the entire stack -- from requirements to database to ETL to BI tools.”
Likewise, the data warehousing team at the University of Illinois found its project backlog growing bigger each year until it reorganized itself into nine small, self-governing interdisciplinary groups. By cross-training its staff and giving members the ability to switch groups every year, the data warehousing team has doubled the number of projects it handles with the same staff.
The Power of One
Netflix goes one step further. Colson believes that even small teams are too slow. “What some people call agile is actually quite slow.” Colson believes that one developer trained in all facets of a BI stack can work faster and more effectively than a team. For example, it’s easier and quicker for one person to decide whether to apply a calculation in the ETL or BI layer than a small team, he says.
Furthermore, Colson doesn’t believe in requirements documents or quality assurance (QA) testing. In fact, he disbanded those groups when he took charge. He believes developers should work directly with users, which is something I posited in a blog titled the Principle of Proximity. He thinks QA testing actually lowers quality because it relieves developers from having to understand the context of the data with which they are working.
It’s safe to say that Colson is not afraid to shake up the establishment. He admits, however, that his approach may not work everywhere: Netflix is a dynamic environment where source systems change daily, so flexibility and fluidity are keys to BI success. He also reports directly to the CEO and has strong support as long as he delivers results.
Both the University of Illinois and Netflix have discovered that agility comes from a flexible organizational model and versatile individuals who have the skills and inclination to deliver complete solutions. They are BI revolutionaries who have successfully unshackled their BI organizations from the bondage of industrial era organizational models and assembly line development processes.
Related Resources
White Papers:
Webinars:
The Growing Practice of Operational Data Integration
April 14, 2010
Speaker: Philip Russom
Developing a Data Quality and Integration Strategy
April 28, 2010
Speaker: Jonathan Geiger
About the Author
Wayne Eckerson is an internationally recognized thought leader in the business intelligence and analytics field. He is a sought-after consultant and noted speaker who thinks critically, writes clearly, and presents persuasively about complex topics.
Eckerson has shared his insights and advice with a wide range of companies. Recent clients include AAA, Air Canada, Carlsberg, Boston Beer, New Balance, Avalon Bay Communities, and TE Connectivity. He has also conducted many groundbreaking research studies, chaired numerous conferences, and written two widely read books: The Secrets of Analytical Leaders: Insights from Information Insiders and Performance Dashboards: Measuring, Monitoring, and Managing Your Business.
Eckerson is founder and principal consultant of Eckerson Group, LLC, a business-technology consulting firm that helps business leaders use data and technology to drive better insights and actions. His team of senior researchers and consultants provide cutting-edge information and advice on business intelligence, analytics, performance management, data governance, data warehousing, and big data. They work closely with organizations that want to assess their current capabilities and develop a strategy for turning data into insights and action. He can be reached at [email protected].