Question and Answer: Business Intelligence Pros Face Tough, Evolving Issues
A new survey reveals that companies still struggle with DW issues and are not leveraging enterprise data warehouses.
- By Linda L. Briggs
- March 3, 2010
In the midst of today’s economic chaos, what types of decisions are businesses making regarding their BI and data warehousing investments? To get answers, we spoke with Vickie Farrell, the market strategy manager for HP’s BI Solutions business unit.
Farrell, who talks at length here about a survey of BI professionals she recently conducted, has spent over 20 years marketing database, data management, and data warehousing products at a variety of companies. She speaks frequently at conferences and has written numerous articles about implementation successes and trends in data warehousing.
BI This Week: You recently surveyed attendees at major BI and data warehousing conferences, including TDWI’s. What was the purpose?
Vickie Farrell: Today’s unusual and uncertain economic environment makes it challenging to predict market trends. HP wanted to augment what we know and hear directly from our clients with some quantitative data about organizations’ directions. We were interested in hearing about their major challenges as well as their investment focus, specifically regarding initiatives they are planning or have in development.
Did you encounter any big surprises in the survey results?
We asked respondents how many BI applications were supported by the enterprise data warehouse at their companies. One surprise was that 54 percent said five or fewer. That’s a very low number.
What do you think this indicates about how organizations are using their enterprise data warehouses?
Considering the investments organizations have made, this result tells us that organizations are not leveraging data warehouses as much as they should be.
It could be that the data warehouses and BI systems implemented in the late 1990s are not able to expand to meet a new generation of analytic applications. That, in fact, may explain another surprising finding from another recent report. HP was a sponsor of the TDWI research report, Next Generation Data Warehouse Platforms, published in early October 2009. In a survey done for that report, 46 percent of respondents said that they anticipate replacing their current primary data warehouse platform by 2012.
To what do you attribute that?
In the survey, respondents identified poor query response, inability to support advanced analytics, inadequate load speed, and inability to scale as the most prevalent problems that would drive them to replace their current platform. What our clients tell us -- even the ones whose DW and BI implementations have been successful in meeting the requirements for which they were originally built -- is that they are struggling to meet new sets of requirements.
Given that, what are some of the biggest challenges companies are facing in data management today?
In our survey, respondents said their companies are struggling with several issues:
- Compliance with increased regulatory requirements
- Support for a broader set of users and applications, including use of BI to support operational decisions
- Integration of data from more sources, including outside the enterprise
- Need for fresher data
- Improved fact-based decision-making and enterprise-wide performance management
Did the survey turn up any other surprises for you?
We asked, “When integrating data from multiple sources for a new application or report, does your company most frequently: access data consolidated in a data warehouse; write explicit code or script; use a manual process; or federate data at runtime?” Despite the investments that have been made in today’s data warehouses, 41 percent said they most frequently write explicit code or use a manual process.
Interestingly, I asked the same question in a survey at the DAMA conference in 2005. At that time, the response was similar: 43 percent. This is clearly a situation that hasn’t improved much in the past few years. However, organizations are now addressing this issue in earnest.
What did your respondents say were their top IT initiatives for BI?
They named data quality, advanced analytics, data governance, and MDM, in that order.
Are these priorities consistent with what you see among HP’s clients?
Yes. There is clearly a growing demand for making more use of advanced analytics, but there is also a recognition that in order to make the most of BI and analytics, you need to get the data right. That means establishing a data governance discipline, achieving and maintaining a level of data quality that is appropriate for the applications, developing a corporate standard for data terms and usage, and providing a much more flexible environment for quickly integrating data from new sources in a way that assures proper semantic reconciliation.
Ultimately, users must have trust in the data such that they can evolve into a community that will use the data to glean better insight and more effectively run their businesses.
Where are most organizations in respect to doing just that?
In the HP survey, we asked about use of several important disciplines, techniques, and technologies. Regarding formal data governance, 29 percent have already implemented data governance, and another 53 percent plan to do so in the next 12 months. Only 18 percent have no plans at all to implement data governance.
Closely related to that, and, in fact, an important step in establishing data governance, is the idea of developing a standard corporate taxonomy for defining business terms. Thirty percent of survey respondents have implemented such a taxonomy already, and another 40-plus percent plan to do so in the next 12 months.
In your report on the survey, you’ve laid out a view of “next-generation” data warehouse and BI goals and demands, along with the challenges most organizations face in getting there. Given the current economic reality, what do you think will be the driving trends in 2010?
We don’t have enough room for detail here, but each January, the HP BI Solutions group publishes our view of the “Top 10 Trends in BI” for the year, available at
www.hp.com/go/bi. One especially interesting item we discuss in the paper is that we believe there are one or two emerging technologies that are reaching enough of a maturity level now to be considered significant in innovative companies’ BI plans in 2010.