AI on Your Lakehouse: Context Comes in Shapes, Not Queries — Zach Blumenfeld, Neo4j
Summary
The presentation introduces a methodology for enhancing AI agent capabilities when querying large lakehouse environments (BigQuery, Databricks, Snowflake). The core argument is that standard vector search or Text2SQL methods often fail due to insufficient global context. The solution involves building three reusable 'graph shapes'—Trees, Communities, and Connections—on top of the data metadata using Neo4j. These shapes create a semantic layer that allows agents to understand complex relationships, patterns, and document hierarchies, enabling them to answer high-level, estate-wide questions.
Key takeaways
-
Context is in Shapes, Not Queries
AI agents need more than just data access; they require a structured understanding of how data elements relate. Graph shapes provide this necessary context to prevent the agent from being 'confidently wrong' when dealing with massive datasets.
-
The Three Core Shapes
10:25
1) **Trees (Table of Contents):** Provides a containment tree structure for unstructured data, allowing agents to navigate document hierarchies and sections. 2) **Communities (Themes):** Surfaces global, unknown patterns by clustering highly interconnected nodes using algorithms like Leiden. 3) **Connections (Semantic Layer):** Builds a metadata graph over structured warehouse schemas, guiding the agent on how tables join without requiring full ETL.
-
Agentic Workflow and Tooling
25:00
The process relies heavily on agents (e.g., Claude Code) utilizing specialized tools like the NeoForj CLY, which allows the agent to write custom Cypher queries based on the provided graph schema/specs.
Technical details
-
Connections Shape (Semantic Layer)
2147s
This shape creates a metadata graph over structured data (e.g., BigQuery schema). It uses NeoCarta to ingest only the metadata, not the raw data itself. This semantic layer guides the agent's Text-to-SQL queries by providing context on column and table relationships, allowing for complex joins without physical ETL into the graph.
-
Trees Shape (Containment)
4525s
This shape models document structure using a deterministic approach. It creates a containment tree reflecting folders, documents, and sections. The system uses hierarchical URIs to maintain order and links, enabling the agent to traverse deep into the document content.
-
Communities Shape (Themes)
This shape identifies natural groupings or 'themes' within a corpus by analyzing interlinking relationships. It utilizes graph algorithms like Leiden community detection, which clusters nodes based on high interconnectivity (cohesion) versus external links. The resulting themes provide an overview of the entire knowledge base.
-
Graph Querying and Agentic Coding
1056s
The system emphasizes using agentic workflows (e.g., Claude Code) with specialized skills like the NeoForj Cypher skill and Graph Data Science (GDS) skill. This allows the agent to write complex, custom graph queries rather than requiring manual query writing.
Mentioned resources
- Graph Academy
- NeoForj Graph Intelligence Platform
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.