# Backtest Your Agent Against Yesterday's Web — Ryan Ahern, Exa

## Executive summary

The presentation introduces Exa, an agentic web search tool, and its critical capability: Exa Snapshot. This feature allows users to version-control search results, enabling the backtesting of AI agents and predictive models against the web as it existed on a specific past date. This capability is vital for mitigating look-ahead bias and ensuring that predictive systems are trained and evaluated using only information available at the time of prediction, which is crucial for reliable system development and deployment.

## Key takeaways

- Agentic Web Search Necessity: AI agents are increasingly using search tools (like Exa) to consult the internet, going beyond their internal knowledge window to find comprehensive answers. This process is more efficient and thorough than traditional human search methods.
- The Problem of Look-Ahead Bias: When building predictive systems (e.g., predicting company acquisitions), using a live search necessarily incorporates the most current information, which can leak future knowledge into the evaluation of past events. This leads to inflated performance metrics.
- Exa Snapshot for Backtesting: Exa Snapshot allows users to execute a full search query (version search) or retrieve content (content versioning) as it appeared on a specific historical date. This enables rigorous backtesting of features and models without future information leakage.

## Technical details

- Exa Search Capabilities: Exa provides search for coding agents and agentic applications, serving as a tool for large language models (LLMs) to consult external information. Exa focuses on the core triangle of quality, cost, and latency, offering sub-300 millisecond search speeds.
- Version Control Mechanism: Exa indexes the entire web (tracking trillions of URLs) and stores versions of content. Users can perform two types of historical queries: 1) Content versioning (retrieving a specific URL as of a past date), and 2) Version search (executing a full search query as it would have been performed in the past).
- Backtesting Methodology: Predictive systems can be tested by constraining the search to a specific historical date (e.g., 1 day, 7 days, or 30 days before an event). This allows engineers to test the predictive power of specific features (e.g., 'executive leadership turnover') using only pre-event information.
- Model Improvement Patterns: Future system design can involve adding agentic features to traditional ML models. The system collects structured features (e.g., adverse media scans, counterparty due diligence) via Exa and feeds them into a traditional ML model for prediction, allowing for controlled, feature-by-feature backtesting.

## Practical implications

- Engineers can design and test predictive features (e.g., financial signals, leadership changes) by constraining web search to a specific historical date, ensuring the model does not rely on future knowledge.
- The capability is useful for building robust due diligence systems (KYC/KYB) or financial forecasting models where historical data integrity is paramount.
- When integrating search into an AI agent, developers can use Exa to ground the model's interface into the outside world, ensuring that the system only searches information available at the time of operation.

## Topics

AI Agents, Web Search, Machine Learning, Time Series Analysis, Feature Engineering, Version Control, Ryan Ahern — Exa, AIE CODE summit

Source: https://www.youtube.com/watch?v=L4dUFimVDKU