Connect AI to Billions of Legal Documents — Simon Eskildsen, turbopuffer & Jacob Lauritzen, Legora
This talk details the architectural challenges of scaling AI-powered legal search (Legora) across billions of documents while meeting stringent regulatory requirements for data residency and physical isolation. The speaker describes a critical failure point in using traditional relational/vector databases (like PostgreSQL/Elasticsearch) with aggressive partitioning, where cache thrashing caused search latency to spike from 100ms P99 to 20 seconds. The solution was adopting TurboPuffer, an object storage-native search engine. TurboPuffer solves these issues by making the 'namespace' the atomic unit of storage, allowing for per-namespace encryption and efficient handling of cold data in object storage, thereby achieving massive latency improvements and cost efficiency.
Key takeaways
-
Scaling Failure in Traditional Databases
18:20
Initial attempts using PostgreSQL (PG vector) and aggressive partitioning (4,000 partitions) failed at scale because mixing hot and cold projects caused the system to thrash the cache, spiking P99 latency from 100 milliseconds to 20 seconds.
-
Object Storage Native Architecture
12:00
TurboPuffer is designed to minimize round trips by writing directly to object storage (e.g., S3). This architecture is ideal for regulated environments because the namespace can be the atomic unit for encryption and physical isolation (e.g., per-bucket, per-key).
-
Handling Cold Data and Isolation
12:00
By making the project the unit of storage (namespace), idle projects can rest in object storage, costing nothing, while still maintaining the ability to enforce physical separation and key control required by large financial institutions.
-
Search Optimization for Memory Hierarchy
16:40
TurboPuffer optimizes search by organizing vector data into a tree structure (like a complex B-tree) rather than navigating a graph. This design minimizes expensive round trips across the memory hierarchy (DRAM $ ightarrow$ NVMe SSD $ ightarrow$ Object Storage).