The unreasonable effectiveness of BM25 for agentic search — Jo Kristian Bergum, Hornet.dev
Summary
The talk argues that the effectiveness of the lexical scoring function BM25 (Best Match 25) is experiencing a major resurgence due to the emergence of powerful LLM agents. While the scoring function itself has not changed, the agent acts as a vastly more sophisticated user, capable of formulating long, specific queries and executing complex search trajectories. This shift necessitates building retrieval infrastructure that treats the context window as a limited resource (like a floppy disc) and leverages primitives like file system workspaces and `grep` for efficient, end-to-end task completion.
Key takeaways
-
The User is the Catalyst for BM25's Return
0:03
LLMs possess general knowledge (entities, companies, dates) that allows them to write queries far more specific and complex than human users. This makes BM25, a 30-year-old lexical function, highly relevant again, moving it beyond a mere baseline.
-
Agentic Search Requires a Defined Loop
Agentic search is defined as search within an agent loop, requiring three components: a capable model (able to use tools/formulate queries), a harness (e.g., tool calling or code mode), and an efficient retrieval engine.
-
Retrieval is Critical for Context Management
0:06
The context window is limited (compared to a floppy disc, ~1.4 MB). Retrieval is necessary to fetch only the required information from massive document sets, preventing quality degradation and enabling end-to-end task completion.
-
Evaluation Must Shift to End-to-End Task Completion
0:08
Traditional evaluation methods based on single-shot queries (like computing NDG) are obsolete. Evaluation must instead focus on whether the model can successfully complete a complex task or answer a riddle-like question using the search tool.
Technical details
-
BM25 Scoring Function
2s
BM25 is a lexical scoring function that calculates the relevance score between a query and a document. It remains a strong, explainable primitive, especially useful for matching literal terms (names, SKUs, zip codes) that embedding models might struggle to encode.
-
Agentic Search Architecture
0s
A robust agentic search system requires a model, a harness (e.g., using code mode to expose retrieval functions), and a scalable retrieval engine capable of handling billions of documents.
-
Context Window Management
7s
The context window is a finite resource. To manage this, advanced systems are recommended to dump retrieved documents into a file system workspace and allow the model to use primitives like `grep` and other trained tools, combining sandbox and retrieval infrastructure.
Mentioned resources
- BrowseCom Plus
- Hornet.dev
- AOL Query Logs
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.