Gauge Your Data Warehousing Maturity
by Wayne Eckerson, Director of Research, TDWI
Many of us have managed data warehousing projects for years. Some have delivered highly strategic systems that are treasured by users and valued by top executives. But others have struggled to sustain interest and funding in their data warehouses even though users are crying out for better, more accurate information.
What separates successful from struggling solutions? How does your data warehousing initiative compare to others in the industry? What will it take to get your data warehouse to the next level?
Many data warehousing managers ask these questions today. Unfortunately, there are no quick or easy answers.1 But to provide some guidance, TDWI has developed a data warehousing maturity model that you can use to benchmark your progress. The model provides a quick way for you to gauge where your data warehousing initiative is now and where it needs to go next.
Illustration 1. Business value increases with each successive stage. Most organizations are currently in the Child and Teenager stages.
Six Stages
The maturity model consists of six stages: Prenatal, Infant, Child, Teenager, Adult, and Sage. Business value increases as the data warehouse moves through each successive stage. (See Illustration 1.)
The stages are defined by a number of characteristics, including scope, analytic structure, executive perceptions, types of analytics, stewardship, funding, technology platform, change management and administration (for which we borrow concepts from SEI’s Capability Maturity Model.) This article will address just a few of these characteristics. Organizations evolve at different rates through these six stages, and each may exhibit characteristics of multiple stages at a given time. As such, no one should expect to move cleanly and precisely from one stage to the next.
However, there are two pivotal points in the evolution of any DW/BI initiative, represented in the model as the “gulf” and the “chasm.” Many DW/BI initiatives stall at these points. They remain stuck with one foot in the past and another in the future, unable to make a clean leap beyond. As a result, they never fully reap the benefits in the successive stages.
The primary way to overcome these obstacles is to change executive perceptions. To cross the “gulf,” executives must recognize that DW/BI is more than just a management reporting system and that the spreadsheets and desktop databases they rely on to run the business are actually undermining productivity and effectiveness. To cross the “chasm”—which is much more difficult—executives must perceive the DW/BI environment as a mission-critical enterprise resource that they own, direct, and fund, not the IT department.
But we’re now getting ahead of ourselves. Let me briefly describe each stage and its major characteristics.
Prenatal Stage—Management Reporting
Most established organizations have management reporting systems that generate a standard set of static reports, which are printed and distributed to large numbers of employees on a regular basis, usually weekly, monthly, or quarterly. Since the reports are hand-coded against legacy systems (or an operational data store), the IT department can’t respond rapidly to requests for custom reports. This erodes IT’s credibility and frustrates users who need quick access to information to do their jobs. This inflexibility is particularly intolerable to business analysts whose job is to crunch numbers on behalf of executives and power users who know their way around corporate information systems. Taking matters into their own hands, these individuals circumvent IT by extracting data from source systems and loading it into spreadsheets or desktop databases. This gives rise to our next stage.
Infant Stage—Spreadmarts
Spreadmarts are spreadsheets or desktop databases that function as surrogate data marts. Each contains a unique set of data, metrics, and rules that do not align with other “spreadmarts,” management reports, or analytical systems. Because spreadsheets are so ubiquitous, cheap, and easy to use, spreadmarts proliferate like weeds—organizations have dozens, if not hundreds or thousands, of these pernicious analytic structures.2
Spreadmarts prevent the organization (or CEO) from getting a clear, consistent picture of the enterprise. But eliminating spreadmarts is difficult because they offer high local control at extremely low costs, making it difficult for organizations to cross the “gulf” between stage one and two. In fact, spreadmart users must sacrifice greater degrees of control over their analytical structures until the final two stages, when new development processes and analytical services align local and enterprise interests. (See Illustration 2.)3
Illustration 2. Spreadmarts are hard to eliminate because they offer maximum local control at minimum cost. Only when organizations enter the final two stages do local control and enterprise value align and accelerate together.
Child Stage—Data Marts
In the Child stage, departments recognize the need to empower all their knowledge workers with timely information and insight, not just the business analysts and executives who primarily benefit from spreadmarts.
A data mart is a shared analytic structure that generally supports a single application area, business process, or department. The departmental team gathers information requirements and tailors each data mart to meet the needs of the members in its group. They then provide knowledge workers with an interactive reporting tool (e.g., OLAP or ad hoc query tool or parameterized reports.) The tool lets knowledge workers drill down or across a dimensional structure to follow trends and gain a deeper insight into events driving the process or tasks they manage.
But data marts often fall prey to the same problems that afflict spreadmarts. Each data mart supports unique definitions and rules and extracts data directly from source systems. These so-called “independent” data marts do a great job of supporting local needs, but their data can’t be aggregated to support cross-departmental analysis. What’s needed is a mechanism to integrate data marts without jeopardizing local autonomy. That’s the hallmark of the Teenager stage.
Teenager Stage—Data Warehouses
After building their third data mart, most departments recognize the need to standardize definitions, rules, and dimensions to avoid an integration nightmare down the road. Standardizing data marts can be done in a centralized or decentralized fashion using one of eight strategies. The most common strategy is to create a central data warehouse with logical “dependent” data marts running in the same database as the data warehouse. This type of data warehouse is commonly referred to as a hub-and-spoke data warehouse.
Interactive Reporting and Analysis. Unlike single-subject data marts, a data warehouse encourages deeper levels of analysis. That’s because users can now submit queries across functional boundaries, such as finance and operations, and gain new insights not possible when data was confined to operational and analytical silos.
To better monitor cross-departmental processes and enterprise value chains, organizations deploy dashboarding applications that support alerting, drill-down paths to detailed reports, distributed queries to pull data from non-warehouse system, and more timely loading of warehouse. These dashboarding applications enable organizations to extend the benefits of business intelligence to more individuals than just technically savvy power users. As a result, executives value DW/BI as a tactical way to improve process efficiency, empower more users with information, and embrace factbased decision making.
Adult Stage—The Enterprise Data Warehouse (EDW)
Although a data warehouse delivers many new benefits, it doesn’t solve the problem of analytic silos. Most organizations today have multiple data warehouses acquired through internal development or mergers or acquisitions. Like spreadmarts and independent data marts, divisional data warehouses contain overlapping and inconsistent data, creating barriers to the free flow of information within and between business groups and the processes they manage.
Integration Machine. In the Adult stage, organizations make a firm commitment to achieve a single version of the truth. Executives view data as a corporate asset that is as valuable as people, equipment, and cash. They anoint one data warehouse as the system of record or build a new enterprise data warehouse (EDW) from scratch. This EDW serves as an “integration machine” that continuously consolidates all other analytic structures into itself. A flexible business intelligence layer finishes the job by integrating data in the EDW with external data that is impractical to load into the EDW for one reason or another (e.g., “real time” data feeds or Web data.) Some organizations that have an acquisitions-based growth strategy use an EDW and BI tools as a prime vehicle to integrate newly acquired organizations.
Stewardship and Scorecards. In the Adult stage, the EDW serves as a strategic enterprise resource for integrating data and supporting mission-critical applications that drive the business. To manage this resource, executives establish a strong stewardship program. Executives assign business people to “own” critical data elements and appoint committees at all levels to guide the development and expansion of the EDW resource. On the analytic side, the organization deploys cascading scorecards to align every worker and business process with corporate strategy. The scorecards often overlay existing dashboard applications, refining existing metrics and prioritizing initiatives and budgets to support strategic goals.
ROI. During the Adult phase, investments in the data warehousing environment begin to pay off. The EDW benefits from economies of scale and a fast-track development process that churns out new mission critical applications rapidly. (See Illustration 3.) In addition, users begin to find new and unexpected uses for the data warehousing environment that developers hadn’t anticipated. This “serendipity of scale” further accelerates ROI.
Illustration 3. A data warehousing environment begins to pay off in terms of ROI in stages 4 and 5.
Sage Stage—BI Services
Once the data warehouse becomes a strategic enterprise resource that drives the business with an ever-growing panoply of mission-critical applications, you may think your job is done. And it may well be! However, there are additional opportunities to increase the strategic value of your EDW by driving the resource outward and downward.
Interactive Extranets. Many companies today are already opening up their data warehouses to customers and suppliers—extending and integrating value chains across organizational boundaries and driving new market opportunities. Next-generation extranet applications will not just provide static reports on account activity. Rather, they will provide customers and suppliers with simple, yet powerful interactive reporting tools to compare and benchmark their activity and performance to other groups across a multitude of dimensions. Some companies, like Owens & Minor, have created new business units to sell data warehousing and information analysis services, and they are transforming their industries as a result.
Web Services. At the same time, EDW development teams are turning analytical data and BI functionality into Web services that developers—both internal and external to the organization—can leverage with proper authorization. The advent of BI services turns the EDW and its applications into a market-wide utility that can be readily embedded into any application. With BI services, workers will no longer have to shift contexts to analyze data. The data, information, and insights they need to do their jobs will be embedded in the operational applications they use on a daily basis.
Decision Engines. These BI services will also make it possible for companies to fully capitalize on their investments in statistical analysis and modeling. They will turn models into “decision engines” embedded in internal and external applications. Workers without any statistical background will feed information into these engines and receive recommendations instantaneously. Today, decision engines already form the basis of several types of powerful applications, including fraud detection, Web personalization, and automated loan approval applications.
Once your DW enters the Sage stage, its value increases exponentially as its visibility declines. As a BI service, the data warehouse and analytic server fade into the background, becoming critical infrastructure that no one thinks about until it stops working due to an outage. Our economy has commoditized innumerable services in the past—such as electricity, sewage, water, transportation, and so on. Insights delivered via BI services are simply the next in line.
Conclusion
Whether you already exhibit the characteristics of a “Sage,” or you are still trying to hurdle the gulf between the Infant and Child stages, this Maturity Model can provide guidance and perspective as you continue your journey. The model can show you where you are, how far you’ve come, and where you need to go. It provides guideposts to help keep you sane and calm amid the chaos and strife we contend with each day.
1 TDWI has written several in-depth reports that can shed light on these issues. I would suggest reading TDWI’s 2003 report Smart Companies in the 21st Century: The Secrets of Building a Successful BI Solution. This 35-page report uses quantitative analysis to shed light on what differentiates successful from struggling data warehousing and business intelligence solutions. You can download this report for free at www.tdwi.org/research.
2 Our latest research shows that organizations have on average 2.1 data warehouses, 6 “independent” data marts, 4.5 ODSs, and 28.5 spreadmarts that they want to consolidate. From In Search of a Single Version of Truth: Strategies for Consolidating Analytical Structures, TDWI, July 2004. You can download the full 32-page report for free at www.tdwi.org/Research/.
3 For tips on controlling the proliferation of spreadmarts, see "Reeling in Spreadmarts," TDWI's Case Studies & Solutions, June 2004.
This article originally appeared in the 11/1/2004 issue of TDWI.