Stop Picking Embedding Models Off The MTEB Leaderboard
Summary
Selecting embedding models requires considering factors beyond MTEB leaderboard scores, including operational cost, query latency, and hardware constraints. The talk details advanced techniques like model/vector quantization (e.g., int8 on CPU, FP16 on GPU), utilizing Matryoshka dimensions, and implementing a two-stage ranking architecture: a cheap first phase followed by an expensive reranker. Proper fine-tuning using tools like Vespa Embed is critical for maximizing performance, while developers must be wary of metric pitfalls like the NDCG blind spot.
Key takeaways
-
MTEB scores are insufficient for production planning
The leaderboard does not account for operational costs, query latency, or hardware-specific quantization requirements. Performance depends heavily on how fast the model generates vectors and how efficiently distance metrics (cosine vs. dot product vs. Hamming) are computed.
-
Quantization must match deployment hardware
3:36
For CPU deployment, int8 quantization is recommended for speed while retaining most precision. For GPU, using an FP16 variant of the model is significantly cheaper and faster than running a native int8 model.
-
Implement cheap first phase + expensive reranker
A scalable search system should use a low-cost initial ranking (e.g., binary vectors with Hamming distance) for millions of documents, reserving more computationally expensive methods (like full float embeddings or cross-encoders) only for the top results.
-
Be cautious of metric blind spots
The NDCG metric can fail to reflect true search improvement if not all documents in the corpus are rated. Teams must ensure their relevance function is consistently applied across all documents being evaluated.
Technical details
-
Quantization Techniques
216s
Model quantization: Use int8 for CPU deployment and FP16 for GPU. Vector quantization: Float32 is standard, but bfloat16 or bit embeddings (e.g., packing 8 bits into an int8) can be used to reduce size with minimal loss.
-
Vector Dimensions
330s
Matryoshka dimensions allow models trained such that earlier vector dimensions are more important. Cutting the full vector (e.g., from 2048 to 512) can sometimes yield better performance than the original model.
-
Query Latency Optimization
414s
The embedder's speed directly impacts query latency. Distance computation efficiency is key: Cosine similarity (baseline), Dot product (if vectors are normalized), and Hamming distance (for bit vectors) offer increasingly faster alternatives.
-
Fine-Tuning Strategies
824s
Use tools like Vespa Embed to tune base models. Training data should ideally consist of pairs (query-to-document), but triplets and hard negatives are recommended for optimal results. LLMs can be used as a judge when labeled search data is scarce.
-
Ranking Architecture
The process involves a stateless layer talking to a data layer. Ranking must be split: Phase 1 (matching/initial score) must be cheap (e.g., BM25 or binary search), while the second phase can afford more expensive methods like cross-encoders.
Mentioned resources
- Vespa Embed
- Hugging Face / Sentence Transformers
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.