Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude Institute
Alex Shaw introduces Harbor, a framework designed for evaluating and optimizing AI agents. The talk argues that agent development should be viewed through the lens of machine learning rather than traditional software engineering. This requires treating agent performance as a 'blackbox artifact' and managing it via empirical evaluation—a process formalized by 'rollouts.' Harbor provides the necessary infrastructure (sandboxes, standardized environments) to execute these complex evaluations in parallel.
Key takeaways
-
Agent Development vs. Software Engineering
5:15
Unlike traditional software engineering where behavior is predictable before execution, agentic coding and AI agents are best treated as blackbox artifacts whose performance requires empirical evaluation (e.g., 'Generated code is best treated as a blackbox artifact').
-
The Role of Rollouts in Agent Evaluation
10:30
Agent evaluation relies on 'rollouts' within sandboxed environments. This process involves passing the sandbox to the agent, collecting a trajectory, and then passing it to a verifier which produces rewards. Harbor standardizes this universal process.
-
Harbor as an Interoperable Standard
12:20
Harbor is presented as a common language and open-source framework for specifying agentic environments, allowing interoperability across different agents, models (e.g., GPT 5.5), and sandboxes to maximize data velocity.
-
Diverse Evaluation Use Cases
17:30
Evaluation can be highly specialized, including assessing how well agents build products (e.g., RampBench), how they use a product's headless mode, or automating internal processes.
