How To Build And Evaluate Search Agents
This talk details three critical areas for building and evaluating advanced search agents: evaluation benchmarks, synthetic data generation, and trajectory analysis. The speaker introduces ORBIT, a novel pipeline for creating complex, multi-hop questions without paid APIs or labeled data. For evaluation, the presentation highlights BrowseComp-Plus as a reproducible benchmark that converts QA tasks into open retrieval problems. Key findings emphasize that both the retriever model and the LLM contribute significantly to accuracy, but efficiency (fewer search calls/turns) is crucial for practical deployment. Finally, trajectory analysis tools like Hawkeye are presented as essential for diagnosing agent failures by visualizing query provenance, topic shifts, and repetition.
Key takeaways
-
Search Agents vs. RAG Architecture
5:45
Agentic search differs from naive Retrieval-Augmented Generation (RAG) because the LLM acts as the main driver, iteratively interacting with a retrieval tool to refine queries before generating a final answer. This iterative process is key.
-
ORBIT for Synthetic Data
17:25
The ORBIT pipeline generates complex, multi-hop questions by describing an entity's properties without naming it (inverting the question). It verifies every question by requiring the agent to confirm claims against source documents.
-
BrowseComp-Plus Reproducibility
22:20
To improve reproducibility, BrowseComp-Plus converts QA tasks into open retrieval problems. This involves providing a web corpus (including hard negatives) and human relevance judgments for each query/answer pair.
-
Efficiency vs. Accuracy in Search Agents
29:20
While high Exact Match (EM) accuracy is good, efficiency—measured by the number of search calls or turns—is equally important. A model with fewer search calls but comparable accuracy is often preferred.
-
Trajectory Analysis Importance
41:00
Relying solely on EM accuracy is insufficient. Analyzing agent trajectories helps identify issues like query repetition, topic shifts, and inefficient looping (e.g., a model taking many turns without reaching the answer).