How to Test Your Agents on Real Tasks With Harbor - Xander Song, Arize AI
Summary
Harbor is an open-source framework designed for evaluating AI agents on realistic tasks using real data within isolated, sandboxed Docker environments. It allows engineers to move from 'shipping based on vibes' to 'shipping based on evidence' by defining tasks, environments, and verifiers. The framework enables rigorous, scalable testing of agent changes by measuring side effects and performance metrics (like tool call count) across multiple stochastic trials.
Key takeaways
-
Harbor Architecture: Tasks, Environments, and Verifiers
Harbor defines tasks using natural language instructions (Markdown) and runs them in isolated Docker environments containing relevant data (e.g., SQLite files, Git repos). Verifiers are scripts that check the intended side effects after the agent runs, producing rewards (e.g., pass/fail). This setup allows testing agent implementations while keeping the task and verifier constant.
-
Scalability and Reliability via Trials and Oracles
10:08
Because LLMs and agents are stochastic, Harbor runs multiple independent 'trials' for each task. To validate the setup before using an agent and spending tokens, an optional 'oracle' (a script that solves the task) can be used to ensure the data and verifiers are correct.
-
Phoenix Integration and Open Standards
28:24
The speaker's team uses Phoenix, an observability platform, which ingests agent traces via OTLP (OpenTelemetry Protocol). Phoenix supports Open Inference, an open standard for semantic conventions, allowing the visualization and versioning of tasks, experiments, and metrics.
-
Optimization Testing and Regression
55:21
Harbor can be used not just for correctness testing, but for optimizing metrics, such as reducing tool calls. By comparing jobs (e.g., 'Opus 5 before' vs. 'Opus 5 after' a prompt change), engineers can quantify the impact of prompt tweaks on system behavior.
Technical details
-
Harbor Framework
128s
Harbor is an open-source framework that orchestrates the evaluation of agents. It requires defining a Task (natural language instructions), an Environment (Docker container with data), and Verifiers (scripts checking side effects). The entire process is managed as a Job, which consists of multiple independent Trials.
-
Data Formats and Tracing
337s
Harbor supports various data types (SQLite files, Git repositories). It uses ATIF (Agent Trajectory Interchange Format), a JSON-based specification, to capture the internal steps and details of an agent's execution for debugging and logging.
-
Phoenix and Observability
1704s
Phoenix is Arize AI's platform for AI observability, ingesting traces via OTLP. It provides core services for managing datasets, experiments, and evaluations, and supports Open Inference, which structures AI application data using open semantic conventions.
-
Execution Model
2507s
Harbor itself is a CLI tool that runs on a host machine (or CI runner) and orchestrates the process by spinning up isolated Docker containers for each trial. This design ensures horizontal scalability and isolation.
Mentioned resources
- Harbor
- Phoenix
- Open Inference
- OpenTelemetry Protocol (OTLP)
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.