Connect AI to Billions of Legal Documents — Simon Eskildsen, turbopuffer & Jacob Lauritzen, Legora
Summary
This talk details the architectural challenges of scaling AI-powered legal search (Legora) across billions of documents while meeting stringent regulatory requirements for data residency and physical isolation. The speaker describes a critical failure point in using traditional relational/vector databases (like PostgreSQL/Elasticsearch) with aggressive partitioning, where cache thrashing caused search latency to spike from 100ms P99 to 20 seconds. The solution was adopting TurboPuffer, an object storage-native search engine. TurboPuffer solves these issues by making the 'namespace' the atomic unit of storage, allowing for per-namespace encryption and efficient handling of cold data in object storage, thereby achieving massive latency improvements and cost efficiency.
Key takeaways
-
Scaling Failure in Traditional Databases
18:20
Initial attempts using PostgreSQL (PG vector) and aggressive partitioning (4,000 partitions) failed at scale because mixing hot and cold projects caused the system to thrash the cache, spiking P99 latency from 100 milliseconds to 20 seconds.
-
Object Storage Native Architecture
12:00
TurboPuffer is designed to minimize round trips by writing directly to object storage (e.g., S3). This architecture is ideal for regulated environments because the namespace can be the atomic unit for encryption and physical isolation (e.g., per-bucket, per-key).
-
Handling Cold Data and Isolation
12:00
By making the project the unit of storage (namespace), idle projects can rest in object storage, costing nothing, while still maintaining the ability to enforce physical separation and key control required by large financial institutions.
-
Search Optimization for Memory Hierarchy
16:40
TurboPuffer optimizes search by organizing vector data into a tree structure (like a complex B-tree) rather than navigating a graph. This design minimizes expensive round trips across the memory hierarchy (DRAM $ ightarrow$ NVMe SSD $ ightarrow$ Object Storage).
Technical details
-
Search Workloads
120s
Legora handles two primary search types: Project Search (search confined to a specific project/client unit of work) and Legal Research (deep, complex queries across massive corpora of laws and regulations).
-
Data Isolation and Encryption
360s
Enterprises require 'physical isolation,' which translates to having their own dedicated database and customer-managed encryption keys. TurboPuffer supports this by allowing each namespace to carry its own encryption key and reside in a separate bucket.
-
Vector Search Optimization
1000s
Instead of navigating a graph (which involves multiple expensive S3 round trips), TurboPuffer organizes vector data into a tree structure (a complex B-tree) to minimize round trips and optimize for the memory hierarchy.
-
Full Text Search (FTS)
1150s
FTS is treated as a hashmap where tokens are keys and sets of document IDs are values. The process minimizes round trips by first downloading relevant dictionary parts and then intersecting compressed lists.
Mentioned resources
- TurboPuffer
- Legora
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.