AI on Your Lakehouse: Context Comes in Shapes, Not Queries — Zach Blumenfeld, Neo4j
The presentation introduces a methodology for enhancing AI agent capabilities when querying large lakehouse environments (BigQuery, Databricks, Snowflake). The core argument is that standard vector search or Text2SQL methods often fail due to insufficient global context. The solution involves building three reusable 'graph shapes'—Trees, Communities, and Connections—on top of the data metadata using Neo4j. These shapes create a semantic layer that allows agents to understand complex relationships, patterns, and document hierarchies, enabling them to answer high-level, estate-wide questions.
Key takeaways
-
Context is in Shapes, Not Queries
AI agents need more than just data access; they require a structured understanding of how data elements relate. Graph shapes provide this necessary context to prevent the agent from being 'confidently wrong' when dealing with massive datasets.
-
The Three Core Shapes
10:25
1) **Trees (Table of Contents):** Provides a containment tree structure for unstructured data, allowing agents to navigate document hierarchies and sections. 2) **Communities (Themes):** Surfaces global, unknown patterns by clustering highly interconnected nodes using algorithms like Leiden. 3) **Connections (Semantic Layer):** Builds a metadata graph over structured warehouse schemas, guiding the agent on how tables join without requiring full ETL.
-
Agentic Workflow and Tooling
25:00
The process relies heavily on agents (e.g., Claude Code) utilizing specialized tools like the NeoForj CLY, which allows the agent to write custom Cypher queries based on the provided graph schema/specs.