Stop Picking Embedding Models Off The MTEB Leaderboard
Selecting embedding models requires considering factors beyond MTEB leaderboard scores, including operational cost, query latency, and hardware constraints. The talk details advanced techniques like model/vector quantization (e.g., int8 on CPU, FP16 on GPU), utilizing Matryoshka dimensions, and implementing a two-stage ranking architecture: a cheap first phase followed by an expensive reranker. Proper fine-tuning using tools like Vespa Embed is critical for maximizing performance, while developers must be wary of metric pitfalls like the NDCG blind spot.
Key takeaways
-
MTEB scores are insufficient for production planning
The leaderboard does not account for operational costs, query latency, or hardware-specific quantization requirements. Performance depends heavily on how fast the model generates vectors and how efficiently distance metrics (cosine vs. dot product vs. Hamming) are computed.
-
Quantization must match deployment hardware
3:36
For CPU deployment, int8 quantization is recommended for speed while retaining most precision. For GPU, using an FP16 variant of the model is significantly cheaper and faster than running a native int8 model.
-
Implement cheap first phase + expensive reranker
A scalable search system should use a low-cost initial ranking (e.g., binary vectors with Hamming distance) for millions of documents, reserving more computationally expensive methods (like full float embeddings or cross-encoders) only for the top results.
-
Be cautious of metric blind spots
The NDCG metric can fail to reflect true search improvement if not all documents in the corpus are rated. Teams must ensure their relevance function is consistently applied across all documents being evaluated.