Hybrid BI: To Centralize or Distribute—That is the Question
By Wayne W. Eckerson, Director, TDWI Research
In an ideal world, your BI architecture and team is managed centrally. You build one architecture with one team located in one place and deliver the output to any group that needs it. This ensures significant economies of scale, avoids redundant effort and infrastructure, and delivers the proverbial “single version of truth.”
Of course, few of us live in the ideal world. Most of us work for organizations that are hopelessly dis-integrated due to countless mergers and acquisitions, corporate restructuring, and corporate “go-to-market” strategies that fluctuate between centralized command-and-control structures and decentralized decision making via local empowerment. In the messy conditions of the real world, the question then becomes: how can you deliver the benefits of centralized BI when your organization and BI architecture are scattered and not aligned?
Distribute to Align
The answer is not as hard as you might think: a distributed approach to BI may be the ideal solution after all. Think about it—distributed BI means that the people developing BI solutions work in the business units or departments they serve. They are closer to the business: they report to the same business unit leader, operate under the same incentive system, and often work in the same building or floor as the business users they support. They may even have lunch with business users or hang out with them after work.
In essence, the BI managers and developers are on the same team as the business users. They are not segregated in an IT group with its own reporting structure, policies and procedures, jargon, incentive systems, and career tracks. They don’t need to go through an exhaustive, clinical approach to gathering requirements because they already “live and breathe” the business. As one BI manager in a distributed organization told me several years ago, “We sit side by side with business people and report into the same leadership. The only difference is that we specialize in the data and they specialize in the business processes.”
By embedding BI in business units, organizations can accelerate development and meet business needs more quickly. They break through the IT bottleneck and backlog of projects that tempt ambitious business leaders to circumvent IT entirely and build their own non-aligned systems. What’s not to like?
Downsides of Distributed BI
Of course, there is a yin to every yang. If you are a BI professional, you know the downsides to distributed BI all too well. When business units are empowered to build their own BI systems (or IT systems for that matter), all hell breaks loose. Every group creates a BI environment in their own image: each defines key entities and metrics differently, and each builds systems and teams that are largely redundant. Business units can’t share information or deliver a consistent view of performance across the enterprise, and overlapping systems undermine efficiency, wasting time and money. In essence, the organization operates as multiple, independent fiefdoms instead of a single enterprise with a unified strategy.
This might not be a bad thing if the business is highly diversified with unique products serving different markets. Yet, most organizations, even highly diversified and decentralized ones, eventually see the need to share consistent information about customers, products, and performance. Usually, this request comes from a strategically minded CEO.
Architectural Standards
We’ve circled back to where we started: how do you create an agile, responsive BI environment—in a large organization with multiple business units—that doesn’t undermine information consistency? The first key is to document architectural standards and best practices that distributed groups can use to establish and grow BI capabilities on their own in conjunction with a corporate group.
This is the role of the much-discussed BI Center of Excellence, which flourishes in more decentralized environments where “coordination” rather than “command-and-control” is the corporate watchword. Obviously, creating a robust set of standards and guidelines requires considerable BI maturity and expertise. One reason that many BI Centers of Excellence fail to take root is that the BI teams implementing them don’t have the requisite BI maturity, or they are using a Center of Excellence as a political Band-Aid to revive a failing BI initiative that doesn’t have strong business backing.
Architectural standards provide guidelines that enable distributed groups to work within a corporate BI architecture without breaking it. These standards range from naming conventions, ETL directory structures, file formats, and error logging procedures, to a choice of ETL and BI tools, to dimensional schema, metric calculations, and entity definitions. In addition, the BI team might revise the methodology published by the organization’s project management office to accommodate the nuances of managing BI projects. They might also write guidelines for negotiating vendor contracts, gathering requirements, establishing service level agreements, and managing production processing environments.
These standards work best when the corporate BI shop does the heavy lifting—creating a central repository of data extracted from the organization’s transaction systems and a dimensional schema that defines key areas of the business common to all business units or departments. The corporate BI shop may also be best positioned to handle production processing in a centralized data center that provides 24x7 monitoring, backup and restore, and disaster recovery—things that most distributed groups quickly recognized are best done centrally.
In other words, distributing BI capabilities effectively requires a strong center to hold it together.
Politics and Incentives
Many BI professionals are skeptical that federating a BI architecture can work. Their big fear is: once you let go of the reins to the sled, how can you make sure it goes in the right direction? In other words, what is to prevent distributed groups from violating architectural and semantic standards and undermining information consistency? Sure, you can configure tools and file systems to ensure that administrators adhere to naming, directory, and other conventions, but eventually it comes down to politics, communication, and incentives. Managing this triad is the third key to making a federated BI environment work successfully.
Business units and departments with robust, ongoing BI environments won’t conform to corporate standards “imposed” on them. A smart BI director will enlist these groups to set the standards in the first place and obtain de facto buy in. Conversely, business units just starting out desperately seek standards and guidelines to facilitate their development efforts; they will embrace your plan and be your biggest supporters. Continuous communication and architectural reviews keep distributed groups aligned with corporate standards and objectives.
Some organizations are successfully implementing hybrid data warehousing environments, which try to balance centralized and distributed control. For example, Intuit has created a centralized data warehousing infrastructure within which it allows business units to create their own data marts. This hybrid approach ensures data consistency and architectural economies of scale while giving business units greater flexibility and control over the data that is of interest to them. eBay applies the same approach but at a more granular level, allowing individual business analysts to create their own data marts within the company’s enterprise data warehouse. Using partitions and workload management, eBay enables analysts to create data marts in the Teradata warehouse, combining data from the warehouse and external sources. This approach has eliminated hundreds of renegade data marts at eBay and accelerated the “time to market” of valuable insights.
These hybrid approaches require considerable education, coordination, and trust among corporate and distributed groups or analysts. When it works, it can be an optimal solution, providing both information consistency and business responsiveness.
Mapping to Organizational Structures
Although there is much to be gained by distributing BI capabilities, there are built-in constraints. Ultimately, your BI architecture needs to mirror your company’s overall organizational structure to succeed. This makes it easier and more culturally acceptable to deal with the trade-offs between consistency and responsiveness on one hand, and economies of scale and business agility on the other.
For example, if your organization operates in a highly centralized fashion, you will be most successful if you establish a strong corporate BI shop that creates and maintains the majority of the BI infrastructure for the organization. The only task you should distribute is the creation of ad hoc reports, which can be outsourced to a network of “super users” who write reports on behalf of colleagues in each department. The downside here is that these shops risk creating project backlogs that frustrate business users and tempt them to circumvent the BI team and BI architecture altogether.
If you work in a highly decentralized company, your corporate BI group exists to serve the BI needs of those business units, which may vary greatly. Some business units may have considerable BI expertise and internal resources and may need very little assistance, while others may need full lifecycle support. The downside here is that there may be no way to ensure information consistency across business units or efficient utilization of resources.
Conclusion: Managing the Trade-offs
Creating an agile BI environment that meets business needs quickly without undermining information consistency is not easy. Pulling these polar opposites together would challenge even the most enlightened Zen master. Distributing BI capabilities can improve BI agility and responsiveness but also can undermine information consistency and increase costs. To resolve the inherent tension, BI teams should:
1. Establish a corporate BI group—or center of excellence— with solid understanding of BI practices;
2. Create architectural standards and best practices that can be adopted by business units and departments when creating their own BI capabilities;
3. Establish a centralized data management infrastructure that creates a single, trusted repository of shared data and manages data center operations to ensure a high degree of data integrity and systems reliability; and
4. Align their BI architectures with their company’s organizational structure, distributing as much as is culturally acceptable.
This article originally appeared in the issue of .