TDWI Experts in: Business Intelligence


  • Revolutionary Business Intelligence: When Agile is Not Fast Enough

    Liberating the Means of Production

    Developers of BI unite! It is time that we liberate the means of BI production from our industrial past.

    Too many BI teams are shackled by outdated modes of industrial organization. In our quest for efficiency, we’ve created rigid fiefdoms of specialization that have hijacked the development process (and frankly, sucked all the enjoyment out of it as well).

    We’ve created an insidious assembly line in which business specialists document user requirements that they throw over the wall to data management specialists who create data models that they throw over the wall to data acquisition specialists who capture and transform data that they throw over the wall to reporting specialists who create reports for end users that they throw over the wall to the support team who helps users understand and troubleshoot reports.

    Flattened BI Teams

    Contrary to standard beliefs, linear development based on specialization is highly inefficient. “Coordination [between BI groups] was killing us,” says Eric Colson, director of BI at Netflix. Colson inherited an industrialized BI team set up by managers who came from a banking environment. The first thing Colson did when he inherited the job was tear down the walls and cross-train everyone on the BI staff. “Everyone now can handle the entire stack -- from requirements to database to ETL to BI tools.”

    Likewise, the data warehousing team at the University of Illinois found its project backlog growing bigger each year until it reorganized itself into nine small, self-governing interdisciplinary groups. By cross-training its staff and giving members the ability to switch groups every year, the data warehousing team has doubled the number of projects it handles with the same staff.

    The Power of One

    Netflix goes one step further. Colson believes that even small teams are too slow. “What some people call agile is actually quite slow.” Colson believes that one developer trained in all facets of a BI stack can work faster and more effectively than a team. For example, it’s easier and quicker for one person to decide whether to apply a calculation in the ETL or BI layer than a small team, he says.

    Furthermore, Colson doesn’t believe in requirements documents or quality assurance (QA) testing. In fact, he disbanded those groups when he took charge. He believes developers should work directly with users, which is something I posited in a blog titled the Principle of Proximity. He thinks QA testing actually lowers quality because it relieves developers from having to understand the context of the data with which they are working.

    It’s safe to say that Colson is not afraid to shake up the establishment. He admits, however, that his approach may not work everywhere: Netflix is a dynamic environment where source systems change daily, so flexibility and fluidity are keys to BI success. He also reports directly to the CEO and has strong support as long as he delivers results.

    Both the University of Illinois and Netflix have discovered that agility comes from a flexible organizational model and versatile individuals who have the skills and inclination to deliver complete solutions. They are BI revolutionaries who have successfully unshackled their BI organizations from the bondage of industrial era organizational models and assembly line development processes.

    Related Resources

    White Papers:

    Operational BI: Expanding BI Through New, Innovative Analytics     Business Intelligence: The Definitive Guide for Midsize Organizations


    The Growing Practice of Operational Data Integration
    April 14, 2010
    Speaker: Philip Russom

    Developing a Data Quality and Integration Strategy
    April 28, 2010
    Speaker: Jonathan Geiger

Vendor Q&A

  • Microstrategy

    Q: What is the biggest problem BI professionals face today?

    A: One of the biggest problems BI professionals face today is meeting the needs of all of their workers, not just their "information workers." Although information workers have increased their usage of business intelligence and dashboards over the years, the overall growth in consumption is still sub-optimal.

    Historically, reports and dashboards have been built and used by managers, analysts, and information workers, but many groups of people within an organization's workforce still don't have their business information needs met. This workforce constitutes groups of people performing similar operational job functions, such as sales personnel, customer service reps, line workers, and store managers.

    These BI users usually have limited access to computer systems and do not have time to "hunt" for data. Often served with nothing at all, they sometimes receive hundreds of pages of printed operational reports and are expected to read through a pile of data to identify and make sense of the information that is relevant to them.

    Although operational reports can be richly informative, they are static in nature and can be overwhelming in volume. Users have no easy way to compare and analyze information across pages or across reports, and as expected, these dense operational reports are often unread and underutilized, and many questions remain unanswered.

    The challenge is in building the right decision systems in a way that is consumable by workers of all types. BI professionals must consider questions of user interface, distribution, and comprehension when selecting and building a solution.

    Answer provided by Matt Ipri, director, Americas marketing at MicroStrategy

Related BI Resources

About TDWI Experts

TDWI Experts is a twice-monthly e-newsletter where BI/DW thought leaders share opinions and ommentary about relevant industry topics and the latest technologies.