Hybrid Business Intelligence Organizations: Managing the Trade-offs between Central and Distributed Development
By Wayne Eckerson, Director, TDWI Research
In an ideal world, your business intelligence (BI) architecture and team are managed centrally. You build an architecture with one team, located in one place, and deliver the output to any group that needs it. This ensures significant economies of scale, avoids redundant effort and infrastructure, and delivers the proverbial "single version of truth."
Of course, few of us live in the ideal world. Most of us work for organizations that are hopelessly disintegrated due to countless mergers and acquisitions, corporate restructuring, and various corporate “go-to-market” strategies that fluctuate between centralized command-and-control structures and decentralized decision making via local empowerment. In the messy conditions of the real world, the question then becomes: How can you deliver the benefits of centralized BI when your organization and BI architecture are scattered and nonaligned?
Distribute to Align
The answer is not as hard as you might think: a distributed approach to BI may be the ideal solution after all. Think about it—distributed BI means that the people developing BI solutions work in the business units or departments they serve. They are closer to the business; they report to the same business unit leader, operate under the same incentive system, and often work in the same building or floor as the business users they support. They may even have lunch with business users or hang out with them after work!
In essence, the BI managers and developers are on the same team as the business users, not segregated in an IT group with its own reporting structure, policies and procedures, jargon, incentive systems, and career tracks. They don’t need to go through an exhaustive, clinical approach to gathering requirements because they already “live and breathe” the business. As one BI manager in a distributed organization told me several years ago, “We sit side by side with business people and report into the same leadership. The only difference is that we specialize in the data and they specialize in the business processes.”
By embedding BI into business units, organizations can accelerate development and meet business needs more quickly. They break through the IT bottleneck and backlog of projects that tempt ambitious business leaders to circumvent IT entirely and build their own nonaligned systems. What’s not to like?
The answer is not as hard as you might think: a distributed approach to BI may be the ideal solution after all.
Downsides of Distributed BI
Of course, there is a yin to every yang. If you are a BI professional, you know the downsides to distributed BI all too well. When business units are empowered to build their own BI systems (or IT systems for that matter), all hell breaks loose. Every group creates a BI environment in its own image; each one defines key entities and metrics differently and builds systems and teams that are largely redundant. Business units can’t share information or deliver a consistent view of performance across the enterprise and overlapping systems undermine efficiency, wasting time and money. The organization operates as multiple, independent fiefdoms instead of a single enterprise with a unified strategy.
This might not be a bad thing if the business is highly diversified with unique products serving different markets. Yet most organizations, even highly diversified and decentralized ones, eventually see the need to share consistent information about customers, products, and performance. Usually, this request comes from a strategically minded CEO.
Architectural Standards
We’ve circled back to where we started: How do you create an agile, responsive BI environment that doesn’t undermine information consistency? The first key is to document architectural standards and best practices that distributed groups can use to establish and grow BI capabilities on their own in conjunction with a corporate group.
This is the role of the much-discussed BI center of excellence, which flourishes in more decentralized environments where “coordination” rather than “command-and-control” is the corporate watchword. Obviously, creating a robust set of standards and guidelines requires a lot of BI maturity and expertise. Many BI centers of excellence fail to take root because the BI teams implementing them don’t have the requisite BI maturity, or they are using a center of excellence as a political Band-Aid to revive a failing BI initiative that doesn’t have strong business backing.
Architectural standards provide guidelines that enable distributed groups to work within a corporate BI architecture without breaking it. These standards include naming conventions, ETL directory structures, file formats, error-logging procedures, a choice of ETL and BI tools, dimensional schema, metric calculations, and entity definitions embedded in a semantic layer. In addition, the BI team might revise the methodology published by the organization’s project management office to accommodate the nuances of managing BI projects. The team might also write guidelines for negotiating vendor contracts, gathering requirements, establishing service-level agreements, and managing production processing environments.
These standards work best when the corporate BI shop does the heavy lifting by creating a central repository of data extracted from the organization’s transaction systems and a dimensional schema that defines key areas of the business common to all business units or departments. The corporate BI shop may also be best positioned to handle production processing in a centralized data center that provides 24/7 monitoring, backup and restore, and disaster recovery—things that most distributed groups quickly recognized are best done in a centralized fashion to save time and money. In other words, distributing BI capabilities effectively requires a strong center to hold it together.
With a core BI infrastructure and semantic layer at their disposal, distributed BI groups then build their own data marts and reports, mingling corporate data and definitions with local data that is germane to only their environments. They can move at their own speed and not have to wait in a queue for the central BI team to get to their project.
Politics and Incentives
Many BI professionals are skeptical that the hybrid BI architecture just described will work. Their big fear is that once you let go of the reins to the sled, how can you make sure it goes in the right direction? In other words, what is to prevent distributed groups from violating architectural and semantic standards and undermining information consistency? Sure, you can configure tools and file systems to ensure administrators adhere to naming, directory, and other conventions, but eventually it comes down to politics, communications, and incentives. Managing this triad is the third key to making a federated BI environment work successfully.
Politics and communication. Business units and departments with robust, ongoing BI environments won’t conform to standards imposed on them from corporate. A smart BI director will enlist these groups to set the standards in the first place and obtain de facto buy-in. Conversely, business units that are just starting out desperately seek standards and guidelines to facilitate their development efforts, so they will embrace your plan and be your biggest supporters. Continuous communication and architectural reviews are needed to keep distributed groups aligned with corporate standards and objectives.
Incentives. It’s also difficult to establish a unified BI architecture in a decentralized organization where each business unit or department is responsible for its own IT services and support. Often, it’s best to leave these groups alone and accept that this is the way the organization wants to do business, despite what executives preach. The only way this will change is for the executive team to start charging each business unit an allocation for BI shared services and establish new incentives that reward business unit and departmental leaders for achieving corporate objectives.
Mapping to Organizational Structures
Although there is much to be gained by distributing BI capabilities, there are built-in constraints. Ultimately, your BI architecture needs to mirror your company’s overall organizational structure to succeed. This makes it easier and more culturally acceptable to deal with the trade-offs between consistency and responsiveness on one hand, and economies of scale and business agility on the other.
For example, if your organization operates in a highly centralized fashion, then you will be most successful if you establish a strong corporate BI shop that creates and maintains the majority of the BI infrastructure for the organization. The only task you should distribute is the creation of ad hoc reports, which can be outsourced to a network of “super users” who write reports on behalf of colleagues in each department. The downside here is that these shops risk creating project backlogs that frustrate business users and tempt them to circumvent the BI team and architecture altogether.
If you work in a highly decentralized company, then your corporate BI group exists to serve the BI needs of those business units, which may vary greatly. Some business units may have lots of BI expertise and internal resources and may need little assistance, while others may require full lifecycle support. However, there may be no way to ensure information consistency across business units, or efficient utilization of resources.
In a hybrid environment where there is a combination of centralized and distributed control or where the balance of power is shifting, the dividing line is not always clear and power struggles often ensue. Typically, the fight is about whether business units can create their own data marts or BI semantic layers. This approach requires considerable coordination and trust among corporate and distributed groups. When it works, it can be an optimal solution, providing both information consistency and business responsiveness; when it fails, it yields the worst of both options.
In short, centralized organizations prize consistency and efficiency more than agility and responsiveness, and decentralized organizations value agility and responsiveness over consistency and economies of scale. To succeed, each BI team needs to identify where it is going to fall on this continuum.
Conclusion: Managing the Trade-offs
Creating an agile BI environment that meets business needs quickly without undermining information consistency is not easy. Pulling these polar opposites together is a challenge for even the most enlightened Zen master. Distributing BI capabilities can improve BI agility and responsiveness, but also undermine information consistency and increase costs.
To resolve the inherent tension, BI teams should (1) establish a corporate BI group with solid understanding of BI practices, (2) create architectural standards and best practices that can be adopted by business units and departments when creating their own BI capabilities, and (3) align their BI architectures with their company’s organizational structure, distributing as much as is culturally acceptable.
Wayne Eckerson is the director of Research at TDWI. He is an industry analyst and the author of Performance Dashboards: Measuring, Monitoring, and Managing Your Business (John Wiley & Sons, 2005). The new edition of his book will be available in November 2010. He can be reached at [email protected].
This article originally appeared in the issue of .