How Multi-Vector Retrieval Works at Scale
Summary
This talk introduces Multi-Vector Retrieval, a critical advancement for building sophisticated AI agents and search systems that move beyond the limitations of single vector embeddings. Single vector approaches (which pool token representations) lose low-level detail, making them ineffective for complex, multi-step agentic queries. Multi-vector methods preserve per-token representation, significantly improving retrieval accuracy, especially in out-of-domain or long-context scenarios. The talk details the technical challenges—namely, massive storage and compute overhead—and presents a solution using sparse multi-vector encoding to make billion-document scale retrieval practical.
Key takeaways
-
Single Vector Limitations for Agents
Single vector embeddings pool token representations into one summary vector, which captures high-level semantics but loses the low-level detail required for precise queries issued by AI agents. This loss of specificity is a theoretical limit that single vectors cannot overcome, even with increased dimensionality.
-
Multi-Vector Solution and Scaling
10:15
Multi-vector embeddings retain one embedding per token instead of pooling them. To manage the resulting storage (10x to 100x increase) and compute overhead, the proposed solution uses sparse multi-vector encoding. This technique approximates MaxSim using random projections, allowing efficient retrieval at scale.
-
Performance Gains in Agentic Retrieval
Multi-vector approaches significantly outperform dense models (e.g., a 100M MultiVector model outperforming an 8B dense model) and standard retrieval methods, achieving higher accuracy at substantially lower cost (e.g., 42% accuracy at one thirteenth of the cost).
-
System Architecture for Scale
20:40
For production readiness, the system separates compute from storage and isolates read/write paths. This architecture allows handling high write throughput (e.g., 70 MB/s) without negatively impacting query latencies, maintaining sub-50ms P99 latency even at billion scales.
Technical details
-
Single vs. Multi-Vector Embeddings
26s
In single vector search, tokens are passed through an encoder model and pooled into one representation. In multi-vector search, the token-level representations are preserved (no pooling), allowing for a more granular capture of relevance between every query token and document token.
-
Sparse Multi-Vector Encoding
790s
To make multi-vector retrieval scalable, the algorithm uses sparse representations derived from random projections. This allows the dot product between sparse query and document vectors to approximate MaxSim efficiently. The cost scales with the number of non-zeros in the sparse vector, not the ambient dimension.
-
Two-Stage Retrieval Process
920s
The process involves two stages: 1) A first stage uses approximate MaxSim (via sparse encoding) to prune a billion documents down to a few hundred promising candidates. 2) A second stage re-ranks these candidates using full, exact MaxSim computation to determine the final top K results.
-
System Scalability and Latency
1240s
The architecture separates compute resources for reads and writes. This design enables high write throughput (70 MB/s) while maintaining low query latencies, achieving sub-50ms P99 latency for sparse search at billion scale.
Mentioned resources
- TopK
- Pinecone
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.