# Why AI Agents Need a Data Harness, Not Just a Lakehouse

## Executive summary

The talk outlines the necessity of a 'data harness'—a modern data foundation optimized for conversational-pace querying by AI agents. Traditional data pipelines built for minutes-to-hours turnaround fail under this rapid load, leading to errors and wasted tokens. The solution requires combining an open lakehouse architecture (using standards like Apache Iceberg and Apache Arrow) with a robust semantic layer, advanced caching (like Columnar Cloud Cache/C3), and federation capabilities to ensure data is accessible, understandable, and performant for AI use cases.

## Key takeaways

- AI Agents Require Subsecond Performance: Conversational analytics moves at a conversational pace. Legacy systems designed for minutes-to-hours turnaround cannot support this speed; subsecond response time is a critical requirement for agentic workflows.
- Three Pillars of AI-Ready Data Foundation: Any modern data foundation must be accessible (seeing the whole estate), understandable (having clear, standardized definitions via a semantic layer), and performant (querying data where it lives without moving or cleaning it).
- Open Standards Prevent Vendor Lock-in: The use of open standards like Apache Iceberg, Apache Arrow, and Apache Polaris ensures that the data platform remains vendor-neutral. This allows users to build performance without sacrificing portability.

## Technical details

- Data Lakehouse Architecture: A lakehouse combines the low-cost, scalable storage of a data lake (e.g., S3) with the reliable asset transactions typically found in a data warehouse. This requires three components: 1) Object Storage (the raw data), 2) A Metadata Catalog (like Apache Polaris or Nessie), and 3) A Metadata Format/Table Format (like Apache Iceberg, which provides schema and history management).
- Performance Data Harness Components: The performance harness relies on: **Semantic Layer:** Consolidating business context into a single layer to prevent AI agents from guessing definitions (e.g., what 'customer' or 'revenue' means). **Federation:** Querying data in place across multiple systems (databases, object storage) without physical centralization. **Caching/C3:** Using techniques like Columnar Cloud Cache (C3) and query plan caching to store frequently accessed data near the compute layer, drastically improving speed.
- Advanced Data Optimization: To improve performance, techniques include: **Clustering:** Grouping similar values in rows for better read efficiency. **Small File Problem Mitigation:** Compacting many small files into fewer, larger ones. **Variant Shredding:** A feature added to Iceberg/Parquet that allows semi-structured data (like JSON) to be efficiently queried within a cell, enabling predicate pushdown on nested data.

## Practical implications

- For build engineers managing data pipelines, the focus must shift from simply moving data to ensuring data quality and context (semantic layer) at runtime.
- Implementing open standards like Apache Iceberg is crucial for building resilient data platforms that are not tied to a single vendor's proprietary format or API.
- Performance optimization should include implementing caching strategies (like C3) and utilizing metadata features (clustering, vacuuming) rather than just increasing compute power.

## Topics

AI Agents, Data Lakehouse, Semantic Layer, Apache Iceberg, Data Federation, Query Optimization, Open Standards, Apache Arrow, Apache Polaris

Source: https://www.youtube.com/watch?v=xR-BiPz45-4