TDWI | Training & Research | Business Intelligence, Analytics, Big Data, Data Warehousing

Defining the Hybrid Data Warehouse

Data warehouse architecture continues to evolve as users modernize their warehouses and as vendor, open source, and cloud provider communities roll out innovative new platforms.

• By Philip Russom
• April 21, 2020

Data warehouse (DW) modernization strategies lead to hybrid architectures.

Replatforming as a modernization strategy. TDWI has seen some of its members (and speakers at TDWI conferences) successfully modernize their data warehouses by migrating the whole thing from one data platform to another. Target platforms are usually new types, such as those based on cloud, NoSQL, or Hadoop. Modernization that focuses on a change of data platform is often called "replatforming" or "rip and replace."

For Further Reading: Data Management Best Practices for Cloud and Hybrid Architectures Four Considerations for CIOs Moving Their Enterprise to a Hybrid Cloud Environment Data Automation: The Heart of Data Warehouse Modernization

The catch with replatforming is that it is an invasive approach that interrupts access to the data warehouse for the users, business processes, and technical processes that rely on it. Furthermore, rip-and-replace strategies erase investments made in the prior platform. Yet replatforming is a viable modernization strategy because it provides a long-term realignment to new business and technology requirements.

Augmentation as a modernization strategy. TDWI more often sees platform modernization handled another way -- a common approach is to migrate some pieces of the warehouse architecture to the cloud while leaving other pieces on premises.

This kind of DW augmentation leaves a mature on-premises data warehouse intact to protect that valuable business investment, and users augment and complement the incumbent warehouse with new platforms and data sets that are cloud-based. Even once one or more new platforms are deployed, many users still maintain the original data platform.

Multiplatform and hybrid architectures often result from modernization strategies. The result of replatforming and augmentation is usually some kind of multiplatform data warehouse. That's where data is physically distributed across old and new platforms. The result is also a hybrid data warehouse, when distributed data spans both on-premises and cloud systems. Synonyms include multiplatform data ecosystem, data warehouse environment, and distributed data architecture.

We've been working with distributed data in data warehouse environments since the 1990s. In the early days of DWs, a centralized instance of a database management system (DBMS) managed most warehouse data, while additional platforms (i.e., multiple instances of DBMSs) were optimized for operational data stores (ODSs) and data marts. Today, that tradition continues, but it's expanded by even more platforms optimized for real-time data, self-service data exploration, specific data domains, and multiple forms of advanced analytics. Data and architectures for warehouses and analytics are more distributed and multiplatform than ever because of increased numbers of optimized platforms and the introduction of new data platform types. With the introduction of new cloud data platforms, distributed and multiplatform data architectures are also hybrid, as seen in the hybrid data warehouse.

Strategies That Guide the Architectural Design of a Hybrid Data Warehouse

As you can see, there are good reasons for creating a hybrid architecture for your modern data warehouse. However, modern data warehouse teams and their colleagues need some rules and strategies to guide the design and use of such architectures. To that end, the modern data warehouse can be compartmentalized in various ways to group data sets and workloads for platform assignment and distribution in hybrid architectures.

Segregate reporting and advanced analytics. These two disciplines are related but have different use cases, users, tool types, forms of information delivery, and -- especially -- data requirements. Satisfying all that with one warehouse platform is difficult or impossible. The trend is to preserve and improve a traditional warehouse on premises for reporting, OLAP, and performance management while new investments in data platforms for advanced analytics and data discovery are made on one or more cloud platforms.

Note that segregating data sets and processing workloads like this is a practical way of matching data and use cases to the right data platform and tools set -- and that's a highly valuable goal. However, the components and platforms may easily become silos unless data is integrated across them and most users and apps are allowed to access them.

Segregate the data warehouse and the data lake. A modern best practice is to preserve the traditional warehouse with its focus on reporting (usually on premises) while creating a new data lake as a very large multipurpose repository for non-reporting categories (increasingly on cloud platforms). Lake use cases typically involve many forms of advanced analytics based on mining, statistics, natural language processing, machine learning, and predictive analytics.

In this hybrid architecture, the warehouse and lake complement each other because the strengths of one fill in for the weaknesses of the other. Again, segregating data sets and workloads like this demands integration and access to avoid the creation of silos.

Deploy your modern data lake in a hybrid architecture, just like the modern warehouse. When a data lake is multitenant, it will need to satisfy the diverse data requirements of many data types and use cases. The breadth of requirements forces most organizations to deploy lake data across multiple data platforms, which in turn may be physically located on multiple premises or multiple clouds. For example, self-service data practices assume support for SQL and the relational paradigm, whereas algorithmic analytics and mining unstructured data assume nonrelational platforms such as Hadoop or NoSQL databases.

Adopt diverse data platforms for diverse use cases and their requirements. Here's the real reason for multiplatform data architectures: Modern data is very diverse and becoming more so in terms of its multiple structures, sources, and latencies -- which in turn drive up the complexity of data's capture, interface types, storage, in-storage processing, metadata management, and new forms of data semantics. Likewise, modern business is increasingly digital and data-driven, which demands that organizations double or triple their software portfolios of analytics and operational applications. Given the extreme diversity of data, its management, and the exploding number of business use cases, it is impossible to satisfy the data requirements of all these scenarios with a single data platform -- or even a short list of platforms.

In response, organizations are eager to deploy multiple, diverse data platforms organized as a hybrid data warehouse. That is because the newly extended software portfolio puts them in a much better position to put the right data on the right platform in the right condition at the right time for the right user performing the right use case.

After all, that's the ultimate goal and noble calling of all data management efforts. A multiplatform and/or hybrid architecture is what it takes today to achieve that highly desirable goal.

For Further Learning

For more information, read the 2019 TDWI Checklist Report: Cloud at Scale for the Modern Data Warehouse. Portions of this article were drawn from that report.