Development Techniques for Creating Analytic Applications
Excerpted from the full March 2005 report. TDWI appreciates the sponsorship of ADVIZOR Solutions, Inc., arcplan, Inc., Business Objects, Microsoft Corporation, MicroStrategy, ProClarity Corporation, and SAP America Inc.
By Wayne W. Eckerson, Director of Research, TDWI
Definition
Business intelligence (BI) professionals throughout the world have one thing in common: no matter how they design or architect their systems, the end result is an analytic application. Yet, this phrase is vague and generally misunderstood by most data warehousing and business intelligence (DW/BI) professionals and their business counterparts.
This confusion exists partly because we, as an industry, tend to focus on the tools, technologies, and architectures that we use to create analytic applications, rather than their output. But if we’ve made the term ambiguous from lack of attention, we’ve also bastardized it by giving it a multiplicity of definitions. Although no definition will satisfy every constituency and industry pundit, we use the following definition in this report:
An analytic application consists of a series of logically integrated, interactive reports, including dashboards and scorecards, that enable a wide range of users to access, analyze, and act on integrated information in the context of the business processes and tasks that they manage in a given domain, such as sales, service, or operations.
Build versus Buy
There is a range of approaches for creating analytic applications. On one end of the spectrum—the “buy” side—organizations purchase packaged analytic applications that require minimal customization and little or no coding across a range of functionality. On the other end—the “build” side—programmers write the entire application from scratch using custom code. Between these two poles are hybrid options that blend both packaged and custom approaches.
Pros and Cons. Many organizations seek to purchase tools or packages to standardize their software investments, accelerate deployment, and reduce total cost of ownership. Unfortunately, many companies end up over-customizing the commercial software, undermining its potential to reduce costs and accelerate deployment. And although building applications from scratch provides complete flexibility, it can become very expensive to keep developers on staff to maintain and extend an application that may already exist commercially.
The Need to Customize. Today, no single development approach delivers a complete analytic application out of the box. Even packaged analytic applications supply only 60 to 80 percent of requisite functionality, depending on users’ requirements, the organization’s existing infrastructure, and the data sources used. Thus, organizations must find ways to customize existing BI packages and tools using a variety of “build” approaches.
While the industry is beginning to coalesce around a buy-and-extend approach to creating analytic applications, some organizations are still committed to build-only or buy-only methodologies.
Buy and Extend. It’s no surprise, then, that the majority of respondents to our survey (52 percent) said their organizations prefer to both build and buy application components—the “buy and extend” approach. Here, organizations purchase a BI tool or analytic package and then customize it.
The remaining 48 percent of respondents took a hard line, preferring to either build or buy analytic applications. Nearly twice as many of these “hard-liners” prefer to build rather than buy (31 percent to 17 percent). This two-to-one ratio mirrors the findings in our September 2002 report, The Rise of Analytic Applications: Build or Buy?
While the industry is beginning to coalesce around a buy-and-extend approach to creating analytic applications, some organizations are still committed to build-only or buy-only methodologies.
Spectrum of Development Techniques
Below is a list of development techniques that organizations use to create analytic applications. Illustration 1 shows the extent to which each can deliver a complete analytic application.
1. Packaged Analytic Application. Delivers 60 to 80 percent of a complete analytic application out of the box, usually in a specific business domain.
2. Packaged Data Marts. Provides an ETL tool, source adapters, a target data model, and source-to-target mappings for specific source systems.
3. BI Starter Kits. Templates in a BI tool that contain style sheets, metrics, reports, and some business logic tailored to a business domain.
4. Microsoft Office Tools. Microsoft Excel, PowerPoint, and Access.
5. BI Tools. Query, reporting, and analysis tools.
6. Analytic Development Environments (ADEs). Graphical, point-and-click development environments for rapidly building analytic applications.
7. Scripting. Lightweight programs that enable developers to quickly customize a BI tool or package’s look and feel or functionality.
8. Portal Integration Kits. APIs and code samples that make it easier to embed BI reports and controls into a third-party portal.
9. BI Software Development Kits (SDKs). A set of documents that describe a BI tool’s API and how to use it.
10. Custom Code. A programming language such as 3GL, 4GL, Web scripts, or SQL used to build or modify applications.
11. Modeling Tools. Tools to create conceptual, logical, and physical models for a data warehouse or data mart.
12. ETL. Extract, transform, and load routines or tools to populate data warehouses and data marts.
Application Completeness. An analytic application comprises many elements. At a high level, these include: (1) look and feel, (2) reports and analytics, (3) analytic server, (4) business logic, (5) data model, and (6) source mappings. These elements must be either built or purchased; even if purchased, developers will need to modify or extend them to meet business requirements.
Clearly, no technique alone creates an analytic application, although packaged applications and custom coding go the farthest toward delivering comprehensive functionality.
Development Techniques in Practice
Primary Technique. Given the spectrum of development techniques for building analytic applications, which techniques or combination of techniques do organizations employ in practice?
Not surprisingly, the largest percentage of respondents (36 percent) selected BI tools as the primary technique for developing their analytic applications. Less than half as many selected custom code (16 percent), Microsoft Office tools (15 percent), and packaged analytic applications (11 percent).
All Techniques. Next, we asked respondents to select all the development techniques they used to create their single largest or “primary” analytic application. Their answers would tell us which techniques organizations employ most often, as well as the combination of techniques in use.
Although organizations prefer to build their primary analytic application with BI tools, on the whole, they use custom coding more than any other development technique. More than half (52 percent) of respondents use custom code when building an analytic application. Next, BI tools were selected by nearly the same percentage as in the previous question (35 percent), followed by Microsoft Office tools (28 percent), packaged analytic applications (22 percent), BI tool SDKs (21 percent), and starter templates (11 percent).
Profile of an Analytic Application. Extrapolating from this data, the “typical” analytic application consists of a BI tool that has been customized using custom code, scripts, or SQL, is written partially against the SDK, and supports Excel as an alternative front-end. (See Illustration 2.)
Development Techniques Used in Primary Analytic Applications
Customization Practices. In practice, organizations are most likely to build analytic applications around a BI tool, but then add substantial amounts of custom code (mostly SQL) to customize and extend the application. Most organizations also customize the BI tool itself, focusing on the GUI, calculations, and navigational elements. Developers also spend significant time customizing ETL mappings and data models in packaged applications. Developers—mainly IT staff and application programmers—make frequent changes to analytic applications, while power users are often enlisted to change the front-end environment.
Analytic Development Environments
There is an emerging category of tools that makes it faster and easier to create custom analytic applications. TDWI calls this new toolset an analytic development environment, or ADE.
An ADE is the analytic counterpart to the integrated development environment, or IDE, which developers have used for years to build operational applications. Examples of the more popular IDEs today are Microsoft Visual Studio.NET, Borland’s JBuilder, Eclipse, IBM’s WebSphere Studio, and BEA’s WebLogic Workshop, to name a few. ADEs are the spiritual heir to IDEs, both in functionality and name.
ADEs promise to accelerate the development of custom-built analytic applications as well as make it easier and faster to customize packaged analytic applications.
A Promising Future. ADEs promise to accelerate the development of custom-built analytic applications as well as make it easier and faster to customize packaged analytic applications. An ADE enables developers to drag and drop analytic components onto a screen to rapidly create analytic applications. More than a report designer, ADEs give developers precise control over the look and feel, functionality, and workflow of an application.
As a result, users will soon be using ADEs to “buy and extend” analytic applications, quickly customizing the last 20 to 40 percent of the front end.
In fact, the drag-and-drop nature of ADEs will further shift development responsibilities from IT developers to power users in the field. With an ADE, a power user can easily modify a packaged analytic application, flesh out a report definition, or create a new application or report from scratch (once IT has established data connections and BI query objects). Thus, ADEs will once and for all get the IT staff out of the business of creating reports so they can focus on what they are best at: building robust data architectures and abstraction layers for end users.
ADE tools will also accelerate the trend towards rapid prototyping. Developers and power users can use an ADE tool in a joint application design session to get immediate feedback from users on data, application screens, metrics, and report designs. This iterative process results in better-designed applications that are delivered more rapidly. Many vendors are shipping ADEs for specific applications to facilitate rapid prototyping. For example, many dashboard and scorecard solutions are ADEs.
Service-Oriented Architecture. The real power behind ADEs comes from the fact that vendors have componentized the functionality of their BI tools. In the past, vendors hard-wired presentation, logic, and data functionality together. But the advent of object-oriented programming and service-oriented architectures has enabled vendors to open up their products, componentizing functionality within a services-oriented framework. The upshot is that ADEs enable developers to create multiple instances of components, store them centrally, and reuse them in other applications. This is a much more efficient way of creating and extending applications.
Clearly, BI vendors have recognized the need to deliver “buy and extend” capabilities. Most are starting to deliver ADEs or ADE-like capabilities. In both cases, the tools provide an easier-to-use authoring environment, which is helping to finally move development out of the hands of professional developers and into the hands of power users and business analysts.
Using ADEs to Build Dashboards and Scorecards
Rapid prototyping using ADEs will soon become the predominant method for building dashboards and scorecards. Already, many BI vendors now offer specialized ADEs for creating dashboards.
Definition. A dashboard or scorecard is a graphical display that compares performance against predefined goals. Most people use these two terms interchangeably, although there is a subtle difference. A dashboard records actual performance or behavior—like an automobile dashboard—while a scorecard measures that performance against objectives or goals. In other words, a dashboard tells you how you are doing, while a scorecard tells how well you’re doing. We will use the term “dashboard” from now on to refer to both types of analysis.
Popularity. Dashboards are increasingly popular; a majority of our survey respondents said their group uses a dashboard as its primary analytic application (31 percent) or has deployed one elsewhere (28 percent). Another 24 percent are currently developing a dashboard or scorecard. Thus, almost three-quarters of the people who took our survey either have a dashboard or scorecard or are developing one.
Dashboard Tools. Most organizations created their dashboards from BI tools (41 percent), followed by custom code (22 percent) and Microsoft Office (13 percent). Only 17 percent have purchased a packaged dashboard solution from a vendor. We expect this percentage to rise in the next few years as more vendors provide robust dashboards solutions. Most dashboards support a range of functionality. In general, dashboards have three levels: they let users drill down from high-level graphical indicators (72 percent) to a table or chart (82 percent) that can be filtered to show different views of data (71 percent) to detailed transaction data (54 percent). (See Illustration 3.)
Advanced dashboard implementations make it easy for executives to add or modify metrics. This ability is not overwhelmingly employed, although power users are enlisted to update dashboards in one-third of organizations. Currently, the IT and application development departments are the most likely candidates to update the dashboard to reflect new requirements. (See Illustration 4.)
Who Updates the Dashboard?
Summary: Dashboards are quickly becoming the primary interface to business intelligence information because they conform to the way the majority of users wish to access, analyze, and act on that information. While most dashboards today are strategic in nature and enterprise in scale, the number and type of users supported indicate that we are still in the early stages of dashboard deployments in most organizations.
Conclusion
The goal of a BI professional is to deliver an effective analytic application that makes it easy for a range of users to access, analyze, and act on information tailored to their business processes and domain. The best analytic applications contain navigational logic that steps users through the process of analyzing and acting on data. Building intuitive analytic applications is not easy. Organizations spend an inordinate amount of time customizing and extending commercial products to meet user requirements.
Fortunately, there is help on the way. Most BI vendors are componentizing their BI tools and exposing them through a graphical interface. TDWI calls these tools analytic development environments (ADEs), and they promise to greatly accelerate development time and reduce costs.
The Future is Clear. Many vendors have released their first-generation ADEs, and many are now tailoring them to dashboards and scorecards, where there is a heightened need to customize the interface, especially for executive-level scorecards. Next-generation BI tools will in fact become ADEs or will be embedded in packaged applications to facilitate customization. This next generation of BI tools will support users’ desire to “buy and extend” existing tools and packages rather than start from scratch.
To download the complete report on Development Techniques for Creating Analytic Applications, as well as prior issues of TDWI’s Report Series, visit http://www.tdwi.org/Research/ReportSeries/index.aspx.