# Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude Institute

## Executive summary

Alex Shaw introduces Harbor, a framework designed for evaluating and optimizing AI agents. The talk argues that agent development should be viewed through the lens of machine learning rather than traditional software engineering. This requires treating agent performance as a 'blackbox artifact' and managing it via empirical evaluation—a process formalized by 'rollouts.' Harbor provides the necessary infrastructure (sandboxes, standardized environments) to execute these complex evaluations in parallel.

## Key takeaways

- Agent Development vs. Software Engineering: Unlike traditional software engineering where behavior is predictable before execution, agentic coding and AI agents are best treated as blackbox artifacts whose performance requires empirical evaluation (e.g., 'Generated code is best treated as a blackbox artifact').
- The Role of Rollouts in Agent Evaluation: Agent evaluation relies on 'rollouts' within sandboxed environments. This process involves passing the sandbox to the agent, collecting a trajectory, and then passing it to a verifier which produces rewards. Harbor standardizes this universal process.
- Harbor as an Interoperable Standard: Harbor is presented as a common language and open-source framework for specifying agentic environments, allowing interoperability across different agents, models (e.g., GPT 5.5), and sandboxes to maximize data velocity.
- Diverse Evaluation Use Cases: Evaluation can be highly specialized, including assessing how well agents build products (e.g., RampBench), how they use a product's headless mode, or automating internal processes.

## Technical details

- Harbor Framework: Harbor is an agent evaluation and RL environment framework. It provides a standardized format for specifying environments, enabling parallel rollouts using any agent/model in any sandbox.
- Agentic Environment Components: A robust evaluation requires three components: an instruction (what to do), a virtual computer/sandbox (where it happens), and a verifier (programmatic tests or rubrics) that determines success.
- Harbor Command Line Usage: Rollouts can be initiated via the command line: `harbor run [benchmark] with [model] and [agent].` Parallelization is key, using commands like `launch` to manage large-scale runs.
- Advanced Use Cases (MapReduce): Harbor supports 'agentic map reduce'—running a high volume of agents on distributed compute sandboxes in parallel, followed by an aggregation step to analyze trajectories or detect patterns.

## Practical implications

- Build teams should adopt a rollout-centered approach to AI development, treating agent performance as an empirical metric rather than deterministic code.
- The ability to define and execute standardized 'evals' (benchmarks) is the primary lever for selecting or training optimal models/agents.
- Utilizing Harbor allows companies to scale evaluations by running thousands of rollouts in parallel across distributed compute resources.

## Topics

AI Agents, Machine Learning Operations (MLOps), Agent Evaluation, Software Architecture, Distributed Computing, Harbor, GitHub: harbor-framework/harbor

Source: https://www.youtube.com/watch?v=jRCpXUjz4CI