The unreasonable effectiveness of BM25 for agentic search — Jo Kristian Bergum, Hornet.dev
The talk argues that the effectiveness of the lexical scoring function BM25 (Best Match 25) is experiencing a major resurgence due to the emergence of powerful LLM agents. While the scoring function itself has not changed, the agent acts as a vastly more sophisticated user, capable of formulating long, specific queries and executing complex search trajectories. This shift necessitates building retrieval infrastructure that treats the context window as a limited resource (like a floppy disc) and leverages primitives like file system workspaces and `grep` for efficient, end-to-end task completion.
Key takeaways
-
The User is the Catalyst for BM25's Return
0:03
LLMs possess general knowledge (entities, companies, dates) that allows them to write queries far more specific and complex than human users. This makes BM25, a 30-year-old lexical function, highly relevant again, moving it beyond a mere baseline.
-
Agentic Search Requires a Defined Loop
Agentic search is defined as search within an agent loop, requiring three components: a capable model (able to use tools/formulate queries), a harness (e.g., tool calling or code mode), and an efficient retrieval engine.
-
Retrieval is Critical for Context Management
0:06
The context window is limited (compared to a floppy disc, ~1.4 MB). Retrieval is necessary to fetch only the required information from massive document sets, preventing quality degradation and enabling end-to-end task completion.
-
Evaluation Must Shift to End-to-End Task Completion
0:08
Traditional evaluation methods based on single-shot queries (like computing NDG) are obsolete. Evaluation must instead focus on whether the model can successfully complete a complex task or answer a riddle-like question using the search tool.