# Can Your AI Analyst Keep a Secret? - Aaron Linsky, Bridgewater & Nikhil Benesch, turbopuffer

## Executive summary

This talk addresses the complex challenge of building an AI analyst that can perform high-recall, low-latency searches over massive, highly permissioned datasets (e.g., 100 million documents). The speakers explain that traditional search methods fail when integrating fine-grained permissions with vector search. They introduce 'native filtering' within TurboPuffer, a technique that combines pre- and post-filtering to achieve fast, scalable results (120ms P90 searches across 400 million vectors) while respecting complex access controls.

## Key takeaways

- The Challenge of Private Data Search: AI analysts require access to both public web data and private, permissioned organizational documents (like emails and reports). Since every document can have unique permissions, standard information retrieval methods struggle to maintain both speed and recall.
- Scalability Limits of Brute Force Search: Searching 100 million documents sequentially, even if cached in memory, is too slow (estimated at 5 seconds or more). Scaling requires specialized indexing techniques.
- Native Filtering for Permissions and Vectors: The optimal solution is 'native filtering,' which combines the benefits of pre-filtering (limiting the search space using permissions) and post-filtering (refining results based on vector similarity) into a single, efficient process.
- Performance Metrics: Using native filtering across 400 million vectors (derived from 100 million documents) achieved a P90 search latency of 120 milliseconds.

## Technical details

- Vector Indexing and Search: Vector embeddings are the internal numeric representation of concepts in LLM space (e.g., 1024 dimensions). A vector index allows search engines to find similar concepts by plotting and zooming into relevant areas of the vector space, drastically reducing the search space from all documents to only the most relevant cluster.
- TurboPuffer Architecture: TurboPuffer is a scalable search engine built on object storage (S3, GCS, Azure Blob). This architecture provides 'S3 economics,' allowing data to reside cheaply on object storage and only being pulled into expensive mediums (like SSD) when queried, enabling cost-effective scaling to terabytes.
- Search Strategies Comparison: Post-filtering (query -> get top vectors -> filter by permission) leads to poor recall. Pre-filtering (filter by permission -> scan all matching docs) is too slow at scale. Native filtering combines both to efficiently search only the relevant, permission-matching clusters.

## Practical implications

- Organizations building AI agents over proprietary data must implement advanced search techniques (like native filtering) to ensure both high recall and low latency while strictly adhering to complex, fine-grained permissions.
- The ability to scale search performance from millions to hundreds of millions of vectors is critical for enterprise AI adoption.
- The solution requires translating complex logical permission structures into a specialized filter language that the search engine can process efficiently under the hood.

## Topics

AI Search, Vector Databases, Information Retrieval, Permissions Management, Scalability Engineering, AIE CODE summit

Source: https://www.youtube.com/watch?v=u_jvuCjuvEk