Question 1

What is a data lakehouse and how does it differ from a data warehouse and a data lake?

Accepted Answer

A data lakehouse is an architecture that combines the structured query performance and data governance of a data warehouse with the flexible, low-cost storage and support for diverse data types characteristic of a data lake. A traditional data warehouse stores structured, processed data optimized for SQL-based BI and reporting — it enforces schema on write, meaning data must conform to a defined structure before it can be stored. A data lake stores raw data in its native format — structured, semi-structured, and unstructured — at low cost, but historically lacked the transactional guarantees, query performance, and governance controls needed for production analytics. The lakehouse bridges this gap by layering a metadata and transaction management layer (such as Delta Lake, Apache Iceberg, or Apache Hudi) on top of cloud object storage, enabling ACID transactions, schema enforcement, and high-performance SQL queries directly on data lake storage. The result is a single platform that can serve BI, data science, machine learning, and generative AI workloads without maintaining separate systems for each. Norbert Kremer, Ph.D. and Richard Winter cover lakehouse architecture in depth at TDWI Transform 2026: https://tdwi.org/events/conferences/anaheim/courses/sunday/arch-all-platforms-and-architectures-for-the-cloud-data-lakehouse.aspx

Question 2

What are open table formats and why do they matter for lakehouse architecture?

Accepted Answer

Open table formats — principally Apache Iceberg, Delta Lake, and Apache Hudi — are the technical layer that makes the lakehouse architecture practical. They sit between cloud object storage (S3, ADLS, GCS) and the query engines that read data, providing transactional consistency, schema evolution, partition management, and time-travel capabilities that raw object storage cannot. Their significance for architecture goes beyond technical features: because they are open standards, they decouple storage from compute, meaning organizations can store data once and query it with multiple engines — Spark, Trino, Flink, Snowflake, BigQuery, and others — without vendor lock-in or data duplication. This interoperability is reshaping competitive dynamics among cloud data platforms, as vendors that previously required proprietary storage formats are now supporting open table formats to remain relevant. Choosing an open table format — and a platform ecosystem that supports it well — is one of the most consequential architectural decisions a data team makes today. Norbert Kremer, Ph.D. and Richard Winter address open table formats and their architectural implications at TDWI Transform 2026: https://tdwi.org/events/conferences/anaheim/courses/sunday/arch-all-platforms-and-architectures-for-the-cloud-data-lakehouse.aspx

Question 3

How do you evaluate and compare cloud data platforms like Snowflake, Databricks, BigQuery, and Microsoft Fabric?

Accepted Answer

Evaluating cloud data platforms requires moving beyond vendor benchmarks and feature checklists to understand how architectural differences translate into real-world performance, cost, and operational complexity for your specific workloads. Key dimensions to assess include: storage-compute separation and how each platform implements it — this directly determines scalability and cost when query volume spikes; query engine architecture and how it handles concurrency, with some platforms struggling significantly under high concurrent user loads; support for diverse workload types including SQL analytics, data science, streaming, and generative AI inferencing; native support for open table formats and interoperability with other tools in your ecosystem; and total cost of operation under realistic workload patterns, which often diverges dramatically from list pricing. Snowflake and Databricks have different architectural roots — Snowflake was built for SQL and BI, Databricks for Spark-based data science — and while both have expanded toward each other's territory, those origins still produce meaningful differences in performance and developer experience for different use case mixes. Microsoft Fabric's integration with the Microsoft 365 ecosystem creates advantages for organizations already heavily invested there. BigQuery's serverless model eliminates cluster management but introduces different cost dynamics. Norbert Kremer, Ph.D. and Richard Winter provide a structured comparative framework at TDWI Transform 2026: https://tdwi.org/events/conferences/anaheim/courses/sunday/arch-all-platforms-and-architectures-for-the-cloud-data-lakehouse.aspx

Question 4

Why is cloud data platform scalability so difficult and costly, and how does generative AI make it worse?

Accepted Answer

Scalability challenges in cloud data platforms arise from a mismatch between how platforms are architected and how analytic workloads actually behave. Analytic workloads are bursty and heterogeneous — a morning reporting rush, an afternoon data science training job, and an overnight batch load all have radically different resource profiles. Platforms that scale compute as a single pool struggle to serve all three efficiently and cost-effectively. Storage-compute separation helps but does not eliminate the problem, because query concurrency limits, metadata management bottlenecks, and data shuffling costs all remain. Generative AI amplifies these challenges significantly. LLM inferencing and embedding generation are computationally intensive workloads that compete with analytics for platform resources. Retrieval-augmented generation (RAG) architectures require the data platform to serve as a low-latency vector search engine in addition to an analytics engine — a role most platforms were not designed for. Fine-tuning and retraining workloads add GPU-intensive batch jobs to the mix. Organizations that have not stress-tested their platform architecture against these combined workloads are likely to encounter both performance failures and billing surprises. Norbert Kremer, Ph.D. and Richard Winter cover platform scalability and AI workload implications at TDWI Transform 2026: https://tdwi.org/events/conferences/anaheim/courses/sunday/arch-all-platforms-and-architectures-for-the-cloud-data-lakehouse.aspx

Question 5

What is near real-time analytics and what architectural requirements does it place on a data platform?

Accepted Answer

Near real-time analytics refers to the ability to query and analyze data within seconds to minutes of it being generated — fast enough to support operational decision-making, fraud detection, customer personalization, and monitoring use cases that batch analytics cannot serve. The architectural requirements are substantially different from those of traditional batch analytics. Data ingestion must be continuous rather than periodic, typically via streaming platforms like Apache Kafka or cloud-native equivalents. The data platform must support low-latency writes without sacrificing query performance — a tension that traditional data warehouse architectures were not designed to manage. Micro-batch and streaming query engines must be integrated with the storage layer. And the governance, lineage, and quality controls that apply to batch data must extend to streaming data, which is significantly harder to implement. Cloud lakehouses address this through streaming ingestion support for open table formats, but implementation complexity remains high and platform support varies considerably. Organizations considering near real-time analytics should evaluate platforms not just on query latency but on end-to-end pipeline latency, streaming ingestion throughput, and the maturity of their streaming SQL support. Norbert Kremer, Ph.D. and Richard Winter cover near real-time analytics requirements at TDWI Transform 2026: https://tdwi.org/events/conferences/anaheim/courses/sunday/arch-all-platforms-and-architectures-for-the-cloud-data-lakehouse.aspx

Question 6

How do you select the right cloud data platform for your organization's workloads?

Accepted Answer

Platform selection is one of the highest-stakes architectural decisions a data organization makes — the switching costs are high, the vendor relationships are long, and the wrong choice can produce years of performance problems and budget overruns. A rigorous selection process starts with workload characterization: document the actual query patterns, data volumes, concurrency requirements, latency expectations, and workload types (SQL analytics, data science, streaming, generative AI) your platform must support, because no single platform excels at all of them equally. Run proof-of-concept tests against your actual data and queries — vendor benchmarks are constructed to favor the vendor and rarely reflect real-world mixed workloads. Evaluate total cost of operation under realistic usage patterns, including storage costs, compute costs at expected concurrency levels, and data egress costs which can be substantial and are frequently overlooked. Assess ecosystem fit: what tools does your team already use, what skills do they have, and how well does the platform integrate with your existing pipelines, BI tools, and ML frameworks? Finally, evaluate the vendor's trajectory — open table format support, AI/ML integration roadmap, and pricing model stability matter as much as current features. Norbert Kremer, Ph.D. and Richard Winter provide a practical platform selection framework at TDWI Transform 2026: https://tdwi.org/events/conferences/anaheim/courses/sunday/arch-all-platforms-and-architectures-for-the-cloud-data-lakehouse.aspx

Question 7

What is the role of in-database AI and analytics, and which cloud platforms support it?

Accepted Answer

In-database AI refers to the execution of machine learning inference, embedding generation, and increasingly LLM-powered functions directly within the data platform — eliminating the need to extract data to an external ML serving layer for every prediction. The advantage is significant: moving the model to the data rather than the data to the model reduces latency, eliminates data transfer costs, simplifies architecture, and enables AI capabilities to be embedded directly in SQL workflows that business users and analysts already know. Leading platforms have expanded in-database AI support substantially: Snowflake Cortex provides LLM functions and ML capabilities directly in SQL; BigQuery ML supports model training and inference inside the database; Databricks integrates with MLflow and supports model serving alongside analytics; Teradata VantageCloud has long supported in-database analytics. The depth of support varies considerably — some platforms offer first-class native ML, others rely on external function integrations that reintroduce data movement. As generative AI becomes a standard component of analytics pipelines, in-database AI support will become a primary platform selection criterion. Norbert Kremer, Ph.D. and Richard Winter cover in-database AI capabilities across leading platforms at TDWI Transform 2026: https://tdwi.org/events/conferences/anaheim/courses/sunday/arch-all-platforms-and-architectures-for-the-cloud-data-lakehouse.aspx

TDWI Transform 2026

S3 Platforms and Architectures for the Cloud Data Lakehouse

You Will Learn

Geared To

Register Online