Executive Summary: Data Warehouse Modernization in the Age of Big Data Analytics
- By Philip Russom, Ph.D.
- March 22, 2016
No matter the vintage or sophistication of your organization’s data warehouse (DW) and the environment around it, it probably needs to be modernized in one or more ways. That’s because DWs and requirements for them continue to evolve. Many users need to get caught up by realigning the DW environment with new business requirements and technology challenges. Once caught up, they need a strategy for continuous modernization.
DW modernization assumes many forms, from server upgrades and tweaks for data models, to adding new platforms into the extended data warehouse environment (DWE), to replacing the primary DW platform. Modernization may involve using features previously untapped, such as in-memory databases, in-database analytics, real-time functions, and data federation or virtualization. Systems integrated with the DW need attention, too. Analytics, reporting, and data integration are also modernizing, and the DW is under pressure to provision data in ways that enable modern end-user practices such as visualization, advanced analytics, data prep, and self-service data access. The arrival of big data has made such provisioning more business critical—and more difficult.
User best practices are also modernizing. For example, the move to agile development methods is one of the strongest trends in data warehousing. Similar trends involve lean, logical, and virtual methods. Modernization also affects users’ skills, staffing, and team structure.
DW modernization is truly happening. This report’s survey says that 76% of DWs are evolving briskly; 89% of respondents say modernization is an opportunity for innovation.
According to the survey, the leading drivers behind DW modernization include realigning the DW with new business goals, increasing DW scale for big data, enabling new analytics applications, and embracing new tools or data types and their attendant practices. The chief beneficiaries of modernization include analytics, business management, and real-time operations. The leading barriers involve problems with governance, staffing, funding, designs, and platforms.
The rise of the multiplatform DWE is an evolution of the DW system architecture. Hence, changes at the system architecture level are the most common form of DW modernization (53% of users surveyed). At one end, this involves simple upgrades and patches for hardware and software servers or tools. At the other end, many organizations are adding new data platforms and analytics tools to their extended DWEs to accommodate massive data volumes, new data types, and new analytics-processing workloads.
Platform types being added to the DWE include those based on columns, appliances, event processing, advanced analytics, and Hadoop; these almost always complement the DW without replacing it. As an extreme measure, roughly half of organizations surveyed plan to rip out their current DW platform and replace it with a bigger and/or more modern one within three or four years. Compared to today, the average DW will be quite different in a few years—and hopefully more “modern” in the sense of bigger and better functionality, scope, speed, scale, user service, and business value. Yet the core platform (whether old or new) will continue to be relational (whether old or new), and new platforms (especially Hadoop) will improve substantially (especially with relational functions at scale).
As you can see, it’s important to modernize a DWE to keep it competitive, relevant, growing, and aligned with new business and technology requirements. User organizations, however, struggle to understand the trends and take the right action. This report presents the many issues and categories of modernization, plus the strategies, methods, and enabling technologies that lead to success.
IBM, Pentaho (a Hitachi Group Company), SAP, SAS, and TimeXtender sponsored the research and writing of this report.
About the Author
Philip Russom, Ph.D., is senior director of TDWI Research for data management and is a well-known figure in data warehousing, integration, and quality, having published over 600 research reports, magazine articles, opinion columns, and speeches over a 20-year period. Before joining TDWI in 2005, Russom was an industry analyst covering data management at Forrester Research and Giga Information Group. He also ran his own business as an independent industry analyst and consultant, was a contributing editor with leading IT magazines, and a product manager at database vendors. His Ph.D. is from Yale. You can reach him by email ([email protected]), on Twitter (twitter.com/prussom), and on LinkedIn (linkedin.com/in/philiprussom).