# Rethinking Environments for Long-Horizon Work — Rayan Garg, Theta Software

## Executive summary

The talk addresses the challenge of defining and measuring 'long horizon work' for AI agents. The core argument is that progress depends less on headline benchmark numbers (like human-equivalent hours) and more on rigorous design of the environment and verifier mechanisms. For build engineering contexts, this means focusing on how tasks involve complex tool coordination (e.g., CI/CD logs, databases), managing state changes, and implementing robust 'judge models' that verify correctness from the final system state rather than just the agent's path.

## Key takeaways

- Defining Long Horizon: Long horizon is a scalar metric, but relying solely on human-equivalent time (e.g., 16 hours via Meter) or model metrics (tokens/steps) is insufficient. The most accurate measure requires considering all variables and the inherent complexity of the task.
- Measuring Model Capability: Model capability should be measured by environment complexity, specifically tool coordination (how many tools are used) and state change complexity. Tasks that can be artificially stretched by chaining unrelated steps do not meaningfully measure model ability.
- The Importance of Verifiers: For complex software domains, deterministic verifiers are often impractical or impossible. The solution is introducing a 'judge model' (or critic) that verifies correctness by examining both the final state of the environment and the entire execution trajectory.
- Addressing Ambiguity: Since real-world tasks are ambiguous, standardized evaluation is difficult. Judges must be designed to handle open-ended solutions rather than requiring a single reference answer or sample trajectory.

## Technical details

- State Verification and Tooling: In complex tasks (e.g., deployment failure analysis), the judge must access the environment's state directly—checking GitHub logs or AWS CloudWatch logs after the agent completes its work—rather than relying solely on tool call history, which is unreliable.
- Trajectory Processing: Due to long and complex trajectories, simply stuffing the path into a context window is insufficient. The trajectory must be made queryable, potentially by using sub-agents or databases to parse out specific phases (e.g., log analysis vs. code writing) for meaningful evaluation.
- Learnability and Reward Signals: The density of the reward signal is critical. Overloading rubrics with too much detail, especially for frontier problems, can cause judges to struggle with consistent application. QA testing (e.g., Gold, no op variance) is necessary.

## Practical implications

- When designing AI environments for complex tasks, prioritize state-based verification (checking the environment's final state) over trajectory-only analysis.
- Model evaluation should focus on measuring complexity related to tool coordination and state changes, rather than just task length or token count.
- Implement 'judge models' that can handle ambiguity by assessing multiple valid solution paths, mitigating the risk of reward hacking.
- Ensure the testing harness for judges is robust enough to scale with increasingly complex environments and tools.

## Topics

AI Agents, Long Horizon Tasks, Environment Design, Verifier Models (Judges), State Space Collapse, Tool Coordination, Meter, Graphana, GitHub / CI/CD, AWS CloudWatch, GDP valer toolbench, Apex agents

Source: https://www.youtube.com/watch?v=2aS7aKoXn64