# How To Build And Evaluate Search Agents

## Executive summary

This talk details three critical areas for building and evaluating advanced search agents: evaluation benchmarks, synthetic data generation, and trajectory analysis. The speaker introduces ORBIT, a novel pipeline for creating complex, multi-hop questions without paid APIs or labeled data. For evaluation, the presentation highlights BrowseComp-Plus as a reproducible benchmark that converts QA tasks into open retrieval problems. Key findings emphasize that both the retriever model and the LLM contribute significantly to accuracy, but efficiency (fewer search calls/turns) is crucial for practical deployment. Finally, trajectory analysis tools like Hawkeye are presented as essential for diagnosing agent failures by visualizing query provenance, topic shifts, and repetition.

## Key takeaways

- Search Agents vs. RAG Architecture: Agentic search differs from naive Retrieval-Augmented Generation (RAG) because the LLM acts as the main driver, iteratively interacting with a retrieval tool to refine queries before generating a final answer. This iterative process is key.
- ORBIT for Synthetic Data: The ORBIT pipeline generates complex, multi-hop questions by describing an entity's properties without naming it (inverting the question). It verifies every question by requiring the agent to confirm claims against source documents.
- BrowseComp-Plus Reproducibility: To improve reproducibility, BrowseComp-Plus converts QA tasks into open retrieval problems. This involves providing a web corpus (including hard negatives) and human relevance judgments for each query/answer pair.
- Efficiency vs. Accuracy in Search Agents: While high Exact Match (EM) accuracy is good, efficiency—measured by the number of search calls or turns—is equally important. A model with fewer search calls but comparable accuracy is often preferred.
- Trajectory Analysis Importance: Relying solely on EM accuracy is insufficient. Analyzing agent trajectories helps identify issues like query repetition, topic shifts, and inefficient looping (e.g., a model taking many turns without reaching the answer).

## Technical details

- Search Agent Architecture: Agentic search involves an iterative process where the LLM uses a retrieval tool to generate queries, receive context, refine its reasoning, and repeat until it can provide a final answer.
- ORBIT Pipeline: A synthetic data pipeline used for bootstrapping retrieval evaluations. It generates complex questions by describing properties without naming the entity, requiring multiple search hops to solve.
- BrowseComp-Plus Benchmark: A reproducible benchmark that converts QA into an open retrieval problem setting. Key metrics include Exact Match (EM) accuracy and recall (number of positive passages retrieved).
- Agentic Search Evaluation Metrics: Evaluation should consider not only EM accuracy but also the number of search calls/turns used, as efficiency is critical. The goal is to find a balance between high accuracy and low resource usage.
- Hawkeye Visualization Tool: A visualization interface for inspecting agentic search trajectories, allowing users to track query provenance, topic shifts, document novelty, and compare multiple agents side-by-side.

## Practical implications

- Building search agents requires moving beyond simple QA benchmarks to complex, multi-hop reasoning tasks.
- Synthetic data generation methods like ORBIT are crucial for developing robust training datasets without relying on expensive APIs or labeled human data.
- Monitoring agent efficiency (search calls/turns) alongside accuracy is necessary for real-world deployment and optimization.

## Topics

Search Agents, Retrieval Augmented Generation (RAG), Benchmark Engineering, Synthetic Data Generation, Agentic AI, ORBIT, Hawkeye, BrowseComp-Plus, MASC (Multilingual Agentic Search Track)

Source: https://www.youtube.com/watch?v=pi-IW4HYJwU