# Catch Agent Regressions Before You Ship: Evals for Managed Deep Agents

## Executive summary

This session details how to implement robust evaluation (evals) for Managed Deep Agents to prevent performance regressions as the agent's capabilities grow. The process involves using Harbor, which ensures each evaluation runs in a fresh container. Evals are structured into an environment (data/state), a job (instruction), and a check (verifier). The workflow is scaffolded using `mda evals init`, which can be automated by handing the task to a coding agent (e.g., Claude Code). Results and traces are managed and monitored in LangSmith, allowing for continuous evaluation and integration into nightly CI pipelines.

## Key takeaways

- Purpose of Evals: Evals serve two primary goals: catching regressions (ensuring changes don't break existing features) and 'hill climbing' (actively improving agent capabilities). The focus is on defining and catching regressions.
- Harbor's Role in Evaluation: Harbor is bundled into Managed Deep Agents and is crucial because it builds an image around the agent, ensuring every eval runs in a fresh container. It also manages test execution to prevent environment pollution.
- Anatomy of an Eval: An evaluation consists of three parts: the environment (the starting data/state), the job (the instruction, defined in a markdown file), and the check (the verifier that determines if the job was completed adequately).
- Automating Eval Scaffolding: The command `mda evals init` scaffolds the necessary files (e.g., `task.md`, `Dockerfile`, tests). Using the `-i` flag allows the work to be handed off to a coding agent, which generates the initial eval suite.
- Production Monitoring: Evals can be managed in production by running them nightly within a CI system. Results are logged into LangSmith, allowing developers to continuously track agent performance and stability.

## Technical details

- Managed Deep Agents & Harbor: Harbor is the toolkit used to build evals for Managed Deep Agents. It handles containerization for every run, which is necessary because agents may interact with the file system or run external commands.
- Eval Structure: Each eval requires an environment (data/state), a job (instruction file), and a check (verifier). The process is scaffolded using `mda evals init`.
- Eval Generation and Automation: The command `mda evals init -i` hands off the eval work to a coding agent (e.g., Claude Code), which uses the LangChain eval engineering skill to produce the necessary task files and test suites.
- Testing Frameworks: Beyond automated test cases, simpler, readable tests can be implemented using standard frameworks like `pytest`. These tests can validate core principles, such as whether the agent cites only real documents or correctly admits when the corpus is insufficient.
- Monitoring and Tracing: All eval runs, including the resulting rewards, are visible in LangSmith under the Data Sets tab. Tracing is managed automatically with deep agents, providing detailed inspection of the agent's execution path.

## Practical implications

- Integrate eval suites into CI/CD pipelines (e.g., running nightly) to ensure agent stability over time.
- Use `mda evals init` to rapidly scaffold the necessary infrastructure for evaluation, minimizing manual setup.
- Leverage coding agents to automate the generation of initial eval suites, accelerating the development cycle.
- Utilize LangSmith for centralized monitoring, allowing developers to inspect traces and track performance metrics (rewards) across multiple evals.

## Topics

Agent Evaluation, LangChain, Deep Agents, CI/CD, Testing, Containerization, Managed Deep Agents, Harbor x LangChain (evaluating agents), LangSmith

Source: https://www.youtube.com/watch?v=pjv-mp5J4hA