# The unreasonable effectiveness of BM25 for agentic search — Jo Kristian Bergum, Hornet.dev

## Executive summary

The talk argues that the effectiveness of the lexical scoring function BM25 (Best Match 25) is experiencing a major resurgence due to the emergence of powerful LLM agents. While the scoring function itself has not changed, the agent acts as a vastly more sophisticated user, capable of formulating long, specific queries and executing complex search trajectories. This shift necessitates building retrieval infrastructure that treats the context window as a limited resource (like a floppy disc) and leverages primitives like file system workspaces and `grep` for efficient, end-to-end task completion.

## Key takeaways

- The User is the Catalyst for BM25's Return: LLMs possess general knowledge (entities, companies, dates) that allows them to write queries far more specific and complex than human users. This makes BM25, a 30-year-old lexical function, highly relevant again, moving it beyond a mere baseline.
- Agentic Search Requires a Defined Loop: Agentic search is defined as search within an agent loop, requiring three components: a capable model (able to use tools/formulate queries), a harness (e.g., tool calling or code mode), and an efficient retrieval engine.
- Retrieval is Critical for Context Management: The context window is limited (compared to a floppy disc, ~1.4 MB). Retrieval is necessary to fetch only the required information from massive document sets, preventing quality degradation and enabling end-to-end task completion.
- Evaluation Must Shift to End-to-End Task Completion: Traditional evaluation methods based on single-shot queries (like computing NDG) are obsolete. Evaluation must instead focus on whether the model can successfully complete a complex task or answer a riddle-like question using the search tool.

## Technical details

- BM25 Scoring Function: BM25 is a lexical scoring function that calculates the relevance score between a query and a document. It remains a strong, explainable primitive, especially useful for matching literal terms (names, SKUs, zip codes) that embedding models might struggle to encode.
- Agentic Search Architecture: A robust agentic search system requires a model, a harness (e.g., using code mode to expose retrieval functions), and a scalable retrieval engine capable of handling billions of documents.
- Context Window Management: The context window is a finite resource. To manage this, advanced systems are recommended to dump retrieved documents into a file system workspace and allow the model to use primitives like `grep` and other trained tools, combining sandbox and retrieval infrastructure.

## Practical implications

- For build engineers, the focus must shift from optimizing the scoring function itself to optimizing the surrounding infrastructure (the 'harness') that manages context and tool use.
- Implementing retrieval should involve treating the retrieved documents as a file system workspace, allowing the LLM to use standard primitives like `grep` for maximum efficiency.
- Evaluation pipelines must be redesigned to test end-to-end task completion rather than relying on single-query metrics.

## Topics

BM25, Agentic Search, Information Retrieval, LLMs, Context Window, Tool Use, Build Infrastructure, BrowseCom Plus, Hornet.dev, AOL Query Logs

Source: https://www.youtube.com/watch?v=fZH97QHHYjY