How To Build And Evaluate Search Agents
Summary
This talk details three critical areas for building and evaluating advanced search agents: evaluation benchmarks, synthetic data generation, and trajectory analysis. The speaker introduces ORBIT, a novel pipeline for creating complex, multi-hop questions without paid APIs or labeled data. For evaluation, the presentation highlights BrowseComp-Plus as a reproducible benchmark that converts QA tasks into open retrieval problems. Key findings emphasize that both the retriever model and the LLM contribute significantly to accuracy, but efficiency (fewer search calls/turns) is crucial for practical deployment. Finally, trajectory analysis tools like Hawkeye are presented as essential for diagnosing agent failures by visualizing query provenance, topic shifts, and repetition.
Key takeaways
-
Search Agents vs. RAG Architecture
5:45
Agentic search differs from naive Retrieval-Augmented Generation (RAG) because the LLM acts as the main driver, iteratively interacting with a retrieval tool to refine queries before generating a final answer. This iterative process is key.
-
ORBIT for Synthetic Data
17:25
The ORBIT pipeline generates complex, multi-hop questions by describing an entity's properties without naming it (inverting the question). It verifies every question by requiring the agent to confirm claims against source documents.
-
BrowseComp-Plus Reproducibility
22:20
To improve reproducibility, BrowseComp-Plus converts QA tasks into open retrieval problems. This involves providing a web corpus (including hard negatives) and human relevance judgments for each query/answer pair.
-
Efficiency vs. Accuracy in Search Agents
29:20
While high Exact Match (EM) accuracy is good, efficiency—measured by the number of search calls or turns—is equally important. A model with fewer search calls but comparable accuracy is often preferred.
-
Trajectory Analysis Importance
41:00
Relying solely on EM accuracy is insufficient. Analyzing agent trajectories helps identify issues like query repetition, topic shifts, and inefficient looping (e.g., a model taking many turns without reaching the answer).
Technical details
-
Search Agent Architecture
345s
Agentic search involves an iterative process where the LLM uses a retrieval tool to generate queries, receive context, refine its reasoning, and repeat until it can provide a final answer.
-
ORBIT Pipeline
1045s
A synthetic data pipeline used for bootstrapping retrieval evaluations. It generates complex questions by describing properties without naming the entity, requiring multiple search hops to solve.
-
BrowseComp-Plus Benchmark
1340s
A reproducible benchmark that converts QA into an open retrieval problem setting. Key metrics include Exact Match (EM) accuracy and recall (number of positive passages retrieved).
-
Agentic Search Evaluation Metrics
1760s
Evaluation should consider not only EM accuracy but also the number of search calls/turns used, as efficiency is critical. The goal is to find a balance between high accuracy and low resource usage.
-
Hawkeye Visualization Tool
2460s
A visualization interface for inspecting agentic search trajectories, allowing users to track query provenance, topic shifts, document novelty, and compare multiple agents side-by-side.
Mentioned resources
- ORBIT
- Hawkeye
- BrowseComp-Plus
- MASC (Multilingual Agentic Search Track)
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.