Can Your AI Analyst Keep a Secret? - Aaron Linsky, Bridgewater & Nikhil Benesch, turbopuffer
Summary
This talk addresses the complex challenge of building an AI analyst that can perform high-recall, low-latency searches over massive, highly permissioned datasets (e.g., 100 million documents). The speakers explain that traditional search methods fail when integrating fine-grained permissions with vector search. They introduce 'native filtering' within TurboPuffer, a technique that combines pre- and post-filtering to achieve fast, scalable results (120ms P90 searches across 400 million vectors) while respecting complex access controls.
Key takeaways
-
The Challenge of Private Data Search
AI analysts require access to both public web data and private, permissioned organizational documents (like emails and reports). Since every document can have unique permissions, standard information retrieval methods struggle to maintain both speed and recall.
-
Scalability Limits of Brute Force Search
13:21
Searching 100 million documents sequentially, even if cached in memory, is too slow (estimated at 5 seconds or more). Scaling requires specialized indexing techniques.
-
Native Filtering for Permissions and Vectors
The optimal solution is 'native filtering,' which combines the benefits of pre-filtering (limiting the search space using permissions) and post-filtering (refining results based on vector similarity) into a single, efficient process.
-
Performance Metrics
Using native filtering across 400 million vectors (derived from 100 million documents) achieved a P90 search latency of 120 milliseconds.
Technical details
-
Vector Indexing and Search
943s
Vector embeddings are the internal numeric representation of concepts in LLM space (e.g., 1024 dimensions). A vector index allows search engines to find similar concepts by plotting and zooming into relevant areas of the vector space, drastically reducing the search space from all documents to only the most relevant cluster.
-
TurboPuffer Architecture
1031s
TurboPuffer is a scalable search engine built on object storage (S3, GCS, Azure Blob). This architecture provides 'S3 economics,' allowing data to reside cheaply on object storage and only being pulled into expensive mediums (like SSD) when queried, enabling cost-effective scaling to terabytes.
-
Search Strategies Comparison
Post-filtering (query -> get top vectors -> filter by permission) leads to poor recall. Pre-filtering (filter by permission -> scan all matching docs) is too slow at scale. Native filtering combines both to efficiently search only the relevant, permission-matching clusters.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.