Data Pipeline Orchestration in a Hybrid Environment
TDWI Speaker: David Loshin, President of Knowledge Integrity
Date: Thursday, August 15, 2019
Time: 9:00 a.m. PT, 12:00 p.m. ET
The desire to modernize data warehousing and analytics environments is influenced by three key trends: the embrace of scalable high performance computing (such as Hadoop and Apache Spark) for platform development, the engagement of “as-a-service” platforms (such as software-as-a-service (SaaS) and platform-as-a-service (PaaS) to support a range of business and data management functions), and migration to the cloud (which provides a broad array of low-cost hosted computing, storage, and application services). The result is a general trend of cloud migration: migrating applications away from traditional on-premises environments toward a variety of different hosted and cloud-based platforms.
Despite the aggressive claims of many cloud platform vendors, there is no magic pill. Application and data migrations are incremental – not all applications can be moved to the cloud at the same time. This means that organizations must operate within a hybrid environment that essentially incorporates data and system services across a variety of platforms including on-premises, a variety of different hosted cloud environments, as well as a growing pool of Internet of Things (IoT) devices and operational systems that stream data. Although the cloud may be a better environment for the long term, it is still complex and will remain so as long there are multiple platform “islands” that demand integration for interoperability. In this webinar we explore the reality of the hybrid environment and its operational complexity and discuss ways that data pipeline orchestration can help to simplify development, migration, and operations within the hybrid environment.
Attendees will learn about:
- Motivations for cloud migration
- The extended information enterprise within the hybrid environment
- Complexity of hybrid environment interoperation and integration
- Data pipelines: operationalization, orchestration, portability
- Workload management
David Loshin