Level: Intermediate to Advanced
Data integration has evolved from traditional batch ETL into data engineering with new concepts and technologies that are proving to be more agile, scalable, and affordable. Understanding the key concepts and relationships of modern data integration will allow data integration teams to evolve into data engineering teams that manage batch ETL, streaming data ingestions, and data pipelines in a single architecture.
This course will have additional emphasis on how to organize and simplify data ingestion environments by exploring how database replication works to stream data from operational databases in a non-intrusive way. We will also look at the DataOps principles and data integration design patterns that should be adopted to deliver the data needed in a portable architecture across on-premises and cloud environments.
You Will Learn
- Essential data ingestion factors including database replication, streams, and managed files
- How streaming data hubs act as a data broker with Apache Kafka
- How to engineer data pipelines with integration design patterns
- How to develop, deploy, and manage data pipelines with DataOps
- Integration architects
- DW/ETL developers
- Data engineers
- Data scientists
- Database administrators
- Data architects