Catch Agent Regressions Before You Ship: Evals for Managed Deep Agents
This session details how to implement robust evaluation (evals) for Managed Deep Agents to prevent performance regressions as the agent's capabilities grow. The process involves using Harbor, which ensures each evaluation runs in a fresh container. Evals are structured into an environment (data/state), a job (instruction), and a check (verifier). The workflow is scaffolded using `mda evals init`, which can be automated by handing the task to a coding agent (e.g., Claude Code). Results and traces are managed and monitored in LangSmith, allowing for continuous evaluation and integration into nightly CI pipelines.
Key takeaways
-
Purpose of Evals
Evals serve two primary goals: catching regressions (ensuring changes don't break existing features) and 'hill climbing' (actively improving agent capabilities). The focus is on defining and catching regressions.
-
Harbor's Role in Evaluation
Harbor is bundled into Managed Deep Agents and is crucial because it builds an image around the agent, ensuring every eval runs in a fresh container. It also manages test execution to prevent environment pollution.
-
Anatomy of an Eval
2:15
An evaluation consists of three parts: the environment (the starting data/state), the job (the instruction, defined in a markdown file), and the check (the verifier that determines if the job was completed adequately).
-
Automating Eval Scaffolding
2:36
The command `mda evals init` scaffolds the necessary files (e.g., `task.md`, `Dockerfile`, tests). Using the `-i` flag allows the work to be handed off to a coding agent, which generates the initial eval suite.
-
Production Monitoring
Evals can be managed in production by running them nightly within a CI system. Results are logged into LangSmith, allowing developers to continuously track agent performance and stability.