# How Multi-Vector Retrieval Works at Scale

## Executive summary

This talk introduces Multi-Vector Retrieval, a critical advancement for building sophisticated AI agents and search systems that move beyond the limitations of single vector embeddings. Single vector approaches (which pool token representations) lose low-level detail, making them ineffective for complex, multi-step agentic queries. Multi-vector methods preserve per-token representation, significantly improving retrieval accuracy, especially in out-of-domain or long-context scenarios. The talk details the technical challenges—namely, massive storage and compute overhead—and presents a solution using sparse multi-vector encoding to make billion-document scale retrieval practical.

## Key takeaways

- Single Vector Limitations for Agents: Single vector embeddings pool token representations into one summary vector, which captures high-level semantics but loses the low-level detail required for precise queries issued by AI agents. This loss of specificity is a theoretical limit that single vectors cannot overcome, even with increased dimensionality.
- Multi-Vector Solution and Scaling: Multi-vector embeddings retain one embedding per token instead of pooling them. To manage the resulting storage (10x to 100x increase) and compute overhead, the proposed solution uses sparse multi-vector encoding. This technique approximates MaxSim using random projections, allowing efficient retrieval at scale.
- Performance Gains in Agentic Retrieval: Multi-vector approaches significantly outperform dense models (e.g., a 100M MultiVector model outperforming an 8B dense model) and standard retrieval methods, achieving higher accuracy at substantially lower cost (e.g., 42% accuracy at one thirteenth of the cost).
- System Architecture for Scale: For production readiness, the system separates compute from storage and isolates read/write paths. This architecture allows handling high write throughput (e.g., 70 MB/s) without negatively impacting query latencies, maintaining sub-50ms P99 latency even at billion scales.

## Technical details

- Single vs. Multi-Vector Embeddings: In single vector search, tokens are passed through an encoder model and pooled into one representation. In multi-vector search, the token-level representations are preserved (no pooling), allowing for a more granular capture of relevance between every query token and document token.
- Sparse Multi-Vector Encoding: To make multi-vector retrieval scalable, the algorithm uses sparse representations derived from random projections. This allows the dot product between sparse query and document vectors to approximate MaxSim efficiently. The cost scales with the number of non-zeros in the sparse vector, not the ambient dimension.
- Two-Stage Retrieval Process: The process involves two stages: 1) A first stage uses approximate MaxSim (via sparse encoding) to prune a billion documents down to a few hundred promising candidates. 2) A second stage re-ranks these candidates using full, exact MaxSim computation to determine the final top K results.
- System Scalability and Latency: The architecture separates compute resources for reads and writes. This design enables high write throughput (70 MB/s) while maintaining low query latencies, achieving sub-50ms P99 latency for sparse search at billion scale.

## Practical implications

- Start by establishing clear evaluation metrics (Evals) on private data to systematically measure retrieval quality.
- Implement relevance tuning by combining content-level relevance scores with user-level signals (e.g., geo distance, popularity) in the scoring function.
- Leverage hybrid search capabilities; multi-vector inherently provides semantic capability while still supporting traditional keyword search (like BM25).
- When implementing, consider optimizing quantization (e.g., 16-bit or 2-bit) and token pruning to manage storage and compute overhead without significant performance loss.

## Topics

Information Retrieval, Vector Databases, AI Agents, Machine Learning Embeddings, Distributed Systems Architecture, TopK, Pinecone

Source: https://www.youtube.com/watch?v=WutkGQEpywA