Why AI Agents Need a Data Harness, Not Just a Lakehouse
The talk outlines the necessity of a 'data harness'—a modern data foundation optimized for conversational-pace querying by AI agents. Traditional data pipelines built for minutes-to-hours turnaround fail under this rapid load, leading to errors and wasted tokens. The solution requires combining an open lakehouse architecture (using standards like Apache Iceberg and Apache Arrow) with a robust semantic layer, advanced caching (like Columnar Cloud Cache/C3), and federation capabilities to ensure data is accessible, understandable, and performant for AI use cases.
Key takeaways
-
AI Agents Require Subsecond Performance
6:32
Conversational analytics moves at a conversational pace. Legacy systems designed for minutes-to-hours turnaround cannot support this speed; subsecond response time is a critical requirement for agentic workflows.
-
Three Pillars of AI-Ready Data Foundation
9:32
Any modern data foundation must be accessible (seeing the whole estate), understandable (having clear, standardized definitions via a semantic layer), and performant (querying data where it lives without moving or cleaning it).
-
Open Standards Prevent Vendor Lock-in
24:12
The use of open standards like Apache Iceberg, Apache Arrow, and Apache Polaris ensures that the data platform remains vendor-neutral. This allows users to build performance without sacrificing portability.