Understanding Business Intelligence
By Wayne Eckerson, Director of Research, TDWIThe “Data Refinery”
TDWI likens business intelligence to a “data refinery.” To understand this analogy, think of an oil refinery. It’s designed to take a raw material—crude oil—and process it into a multiplicity of products such as gasoline, jet fuel, kerosene, and lubricants. In the same way, a BI environment takes another raw material—data—and processes it into a multiplicity of information products. (See illustration 1.)
BI As a Data Refinery
Illustration 1. A BI environment can be thought of as a “data refinery.”
From Data to Information. More specifically, a data warehouse extracts data from multiple transaction or operational systems and integrates and stores the data. For example, a data warehouse might match and merge customer records from five operational systems (e.g. orders, service, sales, shipments, and loyalty programs) into a single file. This extraction and integration process turns data into a new product—information.
From Information to Knowledge. Then, users equipped with analytical tools (e.g. query, reporting, OLAP, and data mining tools) access and analyze the information in the data warehouse. Their analysis identifies trends, patterns, and exceptions. Analytical tools enable users to turn information into knowledge.
From Knowledge to Rules. Armed with these insights, users then create rules from the trends and patterns they’ve discovered. These rules can be simple (e.g., “Order 50 new units whenever inventory falls below 25 units.”) They can be forecasts or “what if” projections based on past trends and working assumptions. Or the rules can be highly complex, generated by statistical algorithms or models. For example, statistically-generated rules can dynamically configure prices in response to changing market conditions, optimize freight hauling schedules in a large carrier network, or determine the best cross-sell opportunities using customer response models.
From Rules to Plans and Action. Users then create plans that implement the rules. For example, marketers create campaigns that define what offers to make to which customers through various channels (e.g. direct mail or email) based on their analysis of customer segments, models that predict how customers will respond to specific offers, and the results of previous campaigns. The plans turn knowledge and rules into action.
Feedback Loop. Once the plan is executed, the cycle repeats itself. Operational systems capture customer responses to the offers or plan and subsequent transactions (e.g. sales.) This data is then extracted by the data warehouse, integrated with other pertinent data, and analyzed by users who evaluate the effectiveness of their plans and refine them accordingly. The cycle then repeats itself.
Learning Organizations
Five-Step Learning Cycle. This virtuous cycle—in essence, capture data, analyze it, plan, act, and review—creates a learning organization that can respond flexibly and nimbly to new events in the marketplace. (See illustration 2.) When organizations repeat this learning cycle, they gain a strong empirical understanding of how their business operates and how its decisions and actions affect the marketplace and vice versa.
The BI Learning Cycle
Illustration 2. BI uses the same five-step learning cycle that humans use every day: capture, analyze, plan, act, review.
This learning cycle embodies the essence of business intelligence, which TDWI defines as: the processes, tools, and technologies required to turn data into information and information into knowledge and plans that drive effective business activity.
The best BI solutions provide robust support for each step in the learning cycle and enhance an organization’s ability to accelerate the cycle to stay ahead of customers and changing market conditions.
Human Learning. In many respects, the best BI solutions are designed to mimic the processes that humans use every day to learn and make judicious decisions. During our lifetime, we experience millions of events that we assimilate, analyze, and turn into rules, whether consciously or not. Each time we apply a “rule,” we get feedback on its validity, which enables us to refine the rules and adapt to changes in our environment. Our “gut instincts” are no more than the unconscious application of rules refined from millions of life experiences.
Business Intelligence Framework
Now that we understand the conceptual basis of business intelligence, let’s explore the components that comprise a BI environment.
BI Component Framework
Illustration 3. At its most basic, a BI environment consists of a data warehousing environment and an analytical environment.
Data Warehousing Environment. This diagram depicts a basic BI environment as two intersecting ovals. (See illustration 3) TDWI calls the left–hand oval the “data warehousing environment.” This is where the technical team spends 60 to 80 percent of its time. Its job is to extract, clean, model, transform, transfer, and load transaction data from one or more operational systems into the data warehouse.
These data warehousing tasks are not easy because operational data is rarely clean, consistent, or easy to integrate. Like archaeologists, the technical team needs to decipher the meaning and validity of thousands of data elements and values in multiple operational systems. They then need to glue everything back together again into a single coherent “model” of the business, much like a paleontologist might reconstruct a life size model of a dinosaur from its bones.
Needless to say, these tasks take a tremendous amount of time and effort and require that technical teams have a deep understanding of business. In fact, no matter how much business savvy the technical team possesses, it still can’t perform this work without step-by-step guidance from key business experts who can interpret the data and define the rules needed to glue it back together.
Once data archaeology and analysis is complete, the technical team loads the integrated data into a data warehouse, which is a relational database optimized for query processing and report generation. Often, the technical team creates a customized subset of the data warehouse, called a data mart, for users in a single department. A data mart can be implemented using a relational database or a specialized multidimensional database that lets users “slice and dice” data by common business dimensions, such as customer, geography, time, and revenues.
Analytical Environment. The right-hand oval in the previous diagram refers to the analytical environment, which is the domain of the business users, who use analytical tools to query, report, analyze, mine, visualize and, most importantly, act on the data in the data warehouse. Since the majority of business users simply want to interact with standard reports, the technical team creates these in advance and puts them on the corporate intranet. Users can either view the report as a static document, filter the report by relevant criteria (e.g. geography, products), or navigate the report (i.e. search, drill down, drill across) to change the view or level of detail. In addition, many organizations are delivering exception-driven reports, such as dashboards or scorecards, which show how performance compares to plan.
For users who want to explore the data in the warehouse in a more ad hoc fashion, the technical team usually creates a catalog of data fields that users can select from to create a query or custom report. This catalog—or meta data layer—shields users from the complexity of the back-end systems and ensures that queries are formed correctly.
BI versus OLTP Systems
Business executives can make the mistake of failing to understand the difference between a BI system and an operational or transaction system, which runs the business on a daily basis (e.g., order entry, inventory, shipping.) Many companies have botched their BI systems by designing them as if they were transaction processing systems.
Design Differences. The key difference is that BI systems adapt to the business whereas transaction systems impose structure on it. (See Table 1.) Since BI solutions are learning systems, they need to adapt continually to the changing concerns of the business. The questions that business people ask today inevitably are different from the ones they will ask tomorrow, next week, or next year.
System Design: OLTP vs. BI Systems
| Transaction systems | Business intelligence systems | | Automate processes | Support decision making | | Designed for efficiency | Designed for effectiveness | | Structure the business | Adapt to the business | | React to events | Anticipate events | | Optimized for transactions | Optimized for queries |
|
Table 1. Basic design differences between transaction systems and BI systems.
In contrast, transaction systems are designed to impose structure on a well-defined business process—such as taking orders—so that it is done the same way every time. Once you design a transaction system, you usually don’t change it unless business practices change.
The opposite is true with a BI system. The more you change it, the better it becomes. And if it doesn’t change to address new questions, it’s not meeting the needs of the business. From a technical perspective, the things that change in a BI system are the data (e.g. new sources and summaries), the data model, the meta data, reports, and applications. It’s also imperative to implement enabling technologies that make it easy to implement changes.
So, the real challenge with a BI solution is how to design and manage a system that always changes. In other words, how do you create an adaptive system? This is certainly not easy, which is why many experts say building a BI system (or data warehouse) is a journey, not a destination.
Types of Data. The dichotomy between transaction systems and BI systems is also evident in the type of data that each manages. (See Table 2.) OLTP systems track current transactions (e.g. debits, credits, and current account balance) and keep little history around (usually only 60 to 90 days of transactions).
In contrast, a BI system maintains years of detailed transactions from multiple OLTP systems. That’s why many veteran BI systems now hold upwards of 50 terabytes. For comparison, one terabyte is roughly equivalent to the text contained in about 1 million books. Moreover, a BI system summarizes this data and applies calculations to create metrics that the business wants to track. To provide fast responses to queries against such large volumes of data requires a different architecture from OLTP systems—one that is optimized for queries rather than transactions.
Types of Data: OLTP vs. BI
| Transaction data | Business intelligence data | | Current | Historical | | Continuously updated | Periodic snapshots | | Source-specific | Integrated | | Application-oriented | Subject-oriented | | Detailed only | Detailed, summarized, & derived | |
Table 2. Transaction and BI systems store very different types of data.
The Move to Real Time. Until recently, BI systems captured transactions by taking periodic “snapshots” of all the data in a transaction system at a certain time of day or week. But now, companies are trying to improve their operational decisions (as opposed to strategic and tactical decisions, discussed below) by analyzing integrated data in a more timely fashion. For example, store managers want to analyze store revenues from yesterday, not last month, and they want to compare their performance against the same day last year and other stores in the chain.1
To support this type of operational decision making, BI systems are beginning to adapt characteristics of transaction systems. BI teams are using “active data warehouses,” “operational data stores,” and middleware (including EAI and Web Services) to capture data in near real time and make it available to business users as soon as possible. Often, firms attach graphical “business dashboards” to near-real-time data feeds so business users can monitor the status of a process or event by watching changes in meters and gauges on the dashboard.
The Analytical Landscape
Does BI Stand for Analytics? Many people believe that BI refers to the analytical environment only. Their perception is shaped by the fact that the only thing business users manipulate to access data and obtain answers to their questions is the analytical tool installed on their desktop or accessible from a Web browser. They don’t necessarily see the data warehousing environment behind the analytical tool. However, as we have shown, BI is much bigger conceptually and architecturally than query-and-reporting and other analytical tools. Business intelligence systems create a learning environment that enables smart organizations to run their businesses more intelligently.
Evolution of Analytics. Nevertheless, a sea change has happened in the analytical environment that is worth noting. Analytical vendors, many of whom have sponsored this report, have evolved considerably in the past decade. Many now offer suites of tools designed to serve every type of analytical need within an organization. Many also have embedded these tools within packaged analytic applications—prebuilt solutions geared to address the analytical requirements of a specific business area such as procurement or supplier performance. Others have focused on delivering vertically integrated suites that combine data integration software with analytical tools and applications. Still others emphasize analytic development platforms for rapidly building custom analytic applications.2
The Landscape for Analytical Tools
Illustration 4. The majority of users (75 percent) use largely predefined reports to determine what happened (reporting or monitoring) in their domain of responsibility.
The illustration above shows the current analytical landscape, focusing primarily on tools, not analytic applications. (See illustration 4.) It depicts four major categories of analytical tools represented by the intersecting circles. Most types of tools play in multiple categories, which is why the circles overlap.
Types of Analysis. The first three domains—report, analyze, and predict—are used to make strategic and tactical decisions. Strategic decisions involve analyzing data for long-term planning purposes (i.e., next quarter or next year) or managing an organization’s progress towards meeting strategic goals and objectives. Balanced scorecards, planning, and budgeting all involve strategic analysis.
Tactical decisions, on the other hand, drive actions that need to take place in the near future (i.e. next week or month). Tactical decisions are more process-driven than strategic. For example, a retail buyer makes tactical decisions when he or she determines which merchandise to buy in what quantities for different stores.
As mentioned earlier, operational decisions need to be made immediately (i.e. today). Traditionally, users query a succession of OLTP systems and merge the results. But the advent of active data warehousing and real-time analytical devices (e.g. dashboards, alerts, agents, and decision engines) make it possible for business users to analyze near-real-time data within the context of historical, integrated data in the BI system. This gives users a much broader, more accurate frame of reference (e.g. year-over-year or seasonal comparisons) for making nitty-gritty operational decisions.
Reporting and Reacting. Not surprisingly, the largest percentage of users (75 percent) use tools in the reporting or reacting domains. Here, users simply view “reports.” These can be static (paper or online), parameterized (showing only selected variables), or interactive (search, drill down, or drill across within a predefined report view.) The reports can also be dynamic dashboards or scorecards that reflect the status of key performance indicators.
Analysis and Prediction. In the analysis domain, business analysts spend a lot of time crunching data to create forecasts or explore the root cause of various problems or industry trends. In the prediction domain, statisticians or analysts are trained in statistical methods. Companies use data mining tools to create predictive and other types of models that drive mission-critical applications, such as identifying fraudulent credit card activity, forecasting when machine parts in an assembly line will fail, and anticipating which customers will respond to a given offer.
Analytical Depth. We know that organizations gain more value from their analytical environments as users move from reporting (“What happened?”) to analysis (“Why did it happen?”) to predictive analysis (“What will happen tomorrow?”) to monitoring (“What just happened?”) However, this doesn’t mean all users within an organization will follow this evolutionary path. It is important the organization as a whole evolve to greater levels of analytic sophistication in order to deliver the most value possible from BI investments.
Analytical Breadth. To derive the full benefit of their analytical environment, companies must also deploy analytical tools broadly, to all knowledge workers as well as customers and suppliers. The benefits of BI accrue in proportion to the number of users in the system. The more broadly the BI system is used, the more benefits it will deliver.
But it’s important to deploy the right tools to the right types of user. Casual users who only want to view standard reports on a weekly basis will become befuddled if you give them a complicated OLAP or data mining tool. Historically, most analytical tools have been geared to power users, not casual users. This is why many organizations have experienced difficulty getting users to use the BI solution.
Although many organizations would like to purchase a single analytical tool or suite, one size does not fit all—at least yet. Vendors of analytical tools have made great strides in recent years to broaden the audience for their products. In three years, it is likely that their analytical suites will offer best-of-breed functionality for each category depicted in the diagram on page 5.
Conclusion
Business intelligence is fast becoming a critical resource in most organizations. It is the “eyes and ears” of organizations. It provides the insight organizations need to guide them through turbulent markets, roller-coaster economies, and aggressive competitors.
Interestingly, many companies are implementing BI without realizing it. That’s because BI is an embedded component of so many business technology initiatives, including business performance management (BPM), customer relationship management (CRM), supply chain management (SCM), and so on. Each of these initiatives requires a single source of clean, integrated data and a host of analytical tools that enable business users to access, analyze, and act on events in their areas of responsibility.
To succeed with BPM, CRM, SCM, or any other technology-driven initiative, organizations need to lay a strong foundation in BI. They need to hire experts who understand that building a BI system is different from building an OLTP system. They need to create a BI architecture that adapts continuously to changing business needs and questions. Organizations that succeed in building strong BI systems will fare better in all other technology initiatives.
1Since operational systems don't retain history or integrate data from other systems, this type of analysis is virtually impossible outside of the BI environment. However, many organizations--unbeknounst to the CEO--employ dozens of analysts whose full-time job more or less is to manually extract and transform disparate data into personal spreadsheets. The proliferation of these spreadsheets--which TDWI calls "spreadmarts"--is an insidious problem at many companies, and one that can only be addressed with a robust enterprise BI solution.
2 For a complete overview of anlaytic applications, see the TDWI report The Rise of Analytic Applications: Build or Buy.
(Excerpted from the TDWI report Smart Companies in the 21st Century: The Secrets of Creating Successful Business Intelligence Solutions, sponsored by Actuate Corporation; arcplan, Inc.; Business Objects; Cognos Inc.; Crystal Decisions, Inc.; Informatica Corporation; MicroStrategy, Inc.; and Oracle Corporation.)