An Introduction to Data Warehouse Modernization
By Philip Russom, Senior Research Director for Data Management, TDWI
As any data warehouse professional can tell you, the average data warehouse (DW) is today evolving, extending, and modernizing, to support new technology and business requirements, as well as to prove its continued relevance in the age of big data and analytics. This process has become known as data warehouse modernization; synonyms include DW augmentation, automation, and optimization. Every user organization and its DW is a unique scenario, so every modernization program is, too. Even so, a few common situations, drivers, and outcomes have arisen.
DW modernization takes many forms.
For example, common scenarios range from software and hardware server upgrades to the periodic addition of new data subjects, sources, tables, and dimensions. However, data types and data velocities are diversifying aggressively, so data modernization progressively involves users’ diversifying their software portfolios to include tools and data platforms built for big data from new sources. As portfolios swell, most data warehouses (DWs) are evolving – or modernizing – into complex and hybrid multi-platform data warehouse environments (DWEs). Though surrounded by complementary systems and tools, the traditional data warehouse is still the primary core of the modern DWE. Even so, a few organizations are decommissioning current data warehouse platforms to replace them with modern ones optimized for today’s requirements in big data, analytics, real-time operation, high-performance, and cost control. No matter what modernization strategy is in play, all require significant adjustments to the logical layers and systems architectures of the extended DWE.
Looking inside the average data warehouse, we see many opportunities for DW professionals to initiate or expand the use of recent technology advancements, such as in-memory processing, in-database analytics, massively parallel processing (MPP), multi-platform federated queries, and Hadoop. Furthermore, there are many new database management systems purpose-built for analytics, based on columns, appliances, graph, MapReduce, NoSQL, and other innovations. Best practices can likewise be modernized by adapting agile, lean, logical, and virtual methods, or by moving to modern team structures, such as the competency center or center of excellence.
Systems outside the DW need modernization, too.
Looking outside the warehouse, multiple disciplines have their own modern innovations that need support from a more modern DW. For example, new business practices need bigger, newer, and fresher data, so the business can compete on analytics, get actionable business value from new big data, and monitor the business in real time. As another example, business intelligence (BI) is experiencing its own modernization right now, and BI needs the DW to provision data for modern BI practices, such as visualization, data exploration, and self service. Likewise, many organizations are complementing their mature investments in online analytic processing (OLAP) with an exploding array of techniques for advanced analytics.
ANNOUNCEMENT
To learn more about modernizing data warehouses and related IT systems, attend my TDWI webinar Data Warehouse Modernization in the age of Big Data Analytics, coming up on April 14, 2016. Register online for the webinar: http://bit.ly/DWMod16
This webinar will quantify trends in data warehouse modernization and catalog technologies that are relevant. It will also document strategies and user best practices for organizing modernization projects. The goal is to help DW professionals and their business counterparts plan the next generation of their data warehouse, in alignment with business goals.
Posted on March 15, 2016