Backtest Your Agent Against Yesterday's Web — Ryan Ahern, Exa
Summary
The presentation introduces Exa, an agentic web search tool, and its critical capability: Exa Snapshot. This feature allows users to version-control search results, enabling the backtesting of AI agents and predictive models against the web as it existed on a specific past date. This capability is vital for mitigating look-ahead bias and ensuring that predictive systems are trained and evaluated using only information available at the time of prediction, which is crucial for reliable system development and deployment.
Key takeaways
-
Agentic Web Search Necessity
AI agents are increasingly using search tools (like Exa) to consult the internet, going beyond their internal knowledge window to find comprehensive answers. This process is more efficient and thorough than traditional human search methods.
-
The Problem of Look-Ahead Bias
24:03
When building predictive systems (e.g., predicting company acquisitions), using a live search necessarily incorporates the most current information, which can leak future knowledge into the evaluation of past events. This leads to inflated performance metrics.
-
Exa Snapshot for Backtesting
17:14
Exa Snapshot allows users to execute a full search query (version search) or retrieve content (content versioning) as it appeared on a specific historical date. This enables rigorous backtesting of features and models without future information leakage.
Technical details
-
Exa Search Capabilities
200s
Exa provides search for coding agents and agentic applications, serving as a tool for large language models (LLMs) to consult external information. Exa focuses on the core triangle of quality, cost, and latency, offering sub-300 millisecond search speeds.
-
Version Control Mechanism
617s
Exa indexes the entire web (tracking trillions of URLs) and stores versions of content. Users can perform two types of historical queries: 1) Content versioning (retrieving a specific URL as of a past date), and 2) Version search (executing a full search query as it would have been performed in the past).
-
Backtesting Methodology
Predictive systems can be tested by constraining the search to a specific historical date (e.g., 1 day, 7 days, or 30 days before an event). This allows engineers to test the predictive power of specific features (e.g., 'executive leadership turnover') using only pre-event information.
-
Model Improvement Patterns
Future system design can involve adding agentic features to traditional ML models. The system collects structured features (e.g., adverse media scans, counterparty due diligence) via Exa and feeds them into a traditional ML model for prediction, allowing for controlled, feature-by-feature backtesting.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.