# Connect AI to Billions of Legal Documents — Simon Eskildsen, turbopuffer & Jacob Lauritzen, Legora

## Executive summary

This talk details the architectural challenges of scaling AI-powered legal search (Legora) across billions of documents while meeting stringent regulatory requirements for data residency and physical isolation. The speaker describes a critical failure point in using traditional relational/vector databases (like PostgreSQL/Elasticsearch) with aggressive partitioning, where cache thrashing caused search latency to spike from 100ms P99 to 20 seconds. The solution was adopting TurboPuffer, an object storage-native search engine. TurboPuffer solves these issues by making the 'namespace' the atomic unit of storage, allowing for per-namespace encryption and efficient handling of cold data in object storage, thereby achieving massive latency improvements and cost efficiency.

## Key takeaways

- Scaling Failure in Traditional Databases: Initial attempts using PostgreSQL (PG vector) and aggressive partitioning (4,000 partitions) failed at scale because mixing hot and cold projects caused the system to thrash the cache, spiking P99 latency from 100 milliseconds to 20 seconds.
- Object Storage Native Architecture: TurboPuffer is designed to minimize round trips by writing directly to object storage (e.g., S3). This architecture is ideal for regulated environments because the namespace can be the atomic unit for encryption and physical isolation (e.g., per-bucket, per-key).
- Handling Cold Data and Isolation: By making the project the unit of storage (namespace), idle projects can rest in object storage, costing nothing, while still maintaining the ability to enforce physical separation and key control required by large financial institutions.
- Search Optimization for Memory Hierarchy: TurboPuffer optimizes search by organizing vector data into a tree structure (like a complex B-tree) rather than navigating a graph. This design minimizes expensive round trips across the memory hierarchy (DRAM $ ightarrow$ NVMe SSD $ ightarrow$ Object Storage).

## Technical details

- Search Workloads: Legora handles two primary search types: Project Search (search confined to a specific project/client unit of work) and Legal Research (deep, complex queries across massive corpora of laws and regulations).
- Data Isolation and Encryption: Enterprises require 'physical isolation,' which translates to having their own dedicated database and customer-managed encryption keys. TurboPuffer supports this by allowing each namespace to carry its own encryption key and reside in a separate bucket.
- Vector Search Optimization: Instead of navigating a graph (which involves multiple expensive S3 round trips), TurboPuffer organizes vector data into a tree structure (a complex B-tree) to minimize round trips and optimize for the memory hierarchy.
- Full Text Search (FTS): FTS is treated as a hashmap where tokens are keys and sets of document IDs are values. The process minimizes round trips by first downloading relevant dictionary parts and then intersecting compressed lists.

## Practical implications

- The architecture is highly suitable for regulated industries (banks, large law firms) that require strict data residency, physical isolation, and granular key management.
- The ability to efficiently store and query the 'long tail' of cold data (idle projects) dramatically improves cost efficiency and scalability compared to traditional partitioned databases.

## Topics

Search Engineering, Object Storage, Vector Databases, Data Residency, AI/ML, TurboPuffer, Legora

Source: https://www.youtube.com/watch?v=V-isu4eTHgw