# How to Test Your Agents on Real Tasks With Harbor - Xander Song, Arize AI

## Executive summary

Harbor is an open-source framework designed for evaluating AI agents on realistic tasks using real data within isolated, sandboxed Docker environments. It allows engineers to move from 'shipping based on vibes' to 'shipping based on evidence' by defining tasks, environments, and verifiers. The framework enables rigorous, scalable testing of agent changes by measuring side effects and performance metrics (like tool call count) across multiple stochastic trials.

## Key takeaways

- Harbor Architecture: Tasks, Environments, and Verifiers: Harbor defines tasks using natural language instructions (Markdown) and runs them in isolated Docker environments containing relevant data (e.g., SQLite files, Git repos). Verifiers are scripts that check the intended side effects after the agent runs, producing rewards (e.g., pass/fail). This setup allows testing agent implementations while keeping the task and verifier constant.
- Scalability and Reliability via Trials and Oracles: Because LLMs and agents are stochastic, Harbor runs multiple independent 'trials' for each task. To validate the setup before using an agent and spending tokens, an optional 'oracle' (a script that solves the task) can be used to ensure the data and verifiers are correct.
- Phoenix Integration and Open Standards: The speaker's team uses Phoenix, an observability platform, which ingests agent traces via OTLP (OpenTelemetry Protocol). Phoenix supports Open Inference, an open standard for semantic conventions, allowing the visualization and versioning of tasks, experiments, and metrics.
- Optimization Testing and Regression: Harbor can be used not just for correctness testing, but for optimizing metrics, such as reducing tool calls. By comparing jobs (e.g., 'Opus 5 before' vs. 'Opus 5 after' a prompt change), engineers can quantify the impact of prompt tweaks on system behavior.

## Technical details

- Harbor Framework: Harbor is an open-source framework that orchestrates the evaluation of agents. It requires defining a Task (natural language instructions), an Environment (Docker container with data), and Verifiers (scripts checking side effects). The entire process is managed as a Job, which consists of multiple independent Trials.
- Data Formats and Tracing: Harbor supports various data types (SQLite files, Git repositories). It uses ATIF (Agent Trajectory Interchange Format), a JSON-based specification, to capture the internal steps and details of an agent's execution for debugging and logging.
- Phoenix and Observability: Phoenix is Arize AI's platform for AI observability, ingesting traces via OTLP. It provides core services for managing datasets, experiments, and evaluations, and supports Open Inference, which structures AI application data using open semantic conventions.
- Execution Model: Harbor itself is a CLI tool that runs on a host machine (or CI runner) and orchestrates the process by spinning up isolated Docker containers for each trial. This design ensures horizontal scalability and isolation.

## Practical implications

- Build Engineers can use Harbor to create robust, evidence-based integration tests for complex, LLM-driven systems, treating the agent as a black box.
- The framework allows for quantitative comparison of agent performance (e.g., comparing prompt changes) by tracking metrics like tool call count and number of turns.
- By using Oracles, teams can validate the entire testing pipeline (data, environment, verifiers) before incurring costs associated with running live agents.
- The modular, containerized nature of Harbor makes it highly compatible with CI/CD pipelines and scalable sandbox providers (e.g., Daytona).

## Topics

AI Agents, LLM Evaluation, Integration Testing, Observability, Docker, CI/CD, Build Engineering, Harbor, Phoenix, Open Inference, OpenTelemetry Protocol (OTLP)

Source: https://www.youtube.com/watch?v=kRJjSOYqW0I