How Multi-Vector Retrieval Works at Scale
This talk introduces Multi-Vector Retrieval, a critical advancement for building sophisticated AI agents and search systems that move beyond the limitations of single vector embeddings. Single vector approaches (which pool token representations) lose low-level detail, making them ineffective for complex, multi-step agentic queries. Multi-vector methods preserve per-token representation, significantly improving retrieval accuracy, especially in out-of-domain or long-context scenarios. The talk details the technical challenges—namely, massive storage and compute overhead—and presents a solution using sparse multi-vector encoding to make billion-document scale retrieval practical.
Key takeaways
-
Single Vector Limitations for Agents
Single vector embeddings pool token representations into one summary vector, which captures high-level semantics but loses the low-level detail required for precise queries issued by AI agents. This loss of specificity is a theoretical limit that single vectors cannot overcome, even with increased dimensionality.
-
Multi-Vector Solution and Scaling
10:15
Multi-vector embeddings retain one embedding per token instead of pooling them. To manage the resulting storage (10x to 100x increase) and compute overhead, the proposed solution uses sparse multi-vector encoding. This technique approximates MaxSim using random projections, allowing efficient retrieval at scale.
-
Performance Gains in Agentic Retrieval
Multi-vector approaches significantly outperform dense models (e.g., a 100M MultiVector model outperforming an 8B dense model) and standard retrieval methods, achieving higher accuracy at substantially lower cost (e.g., 42% accuracy at one thirteenth of the cost).
-
System Architecture for Scale
20:40
For production readiness, the system separates compute from storage and isolates read/write paths. This architecture allows handling high write throughput (e.g., 70 MB/s) without negatively impacting query latencies, maintaining sub-50ms P99 latency even at billion scales.