# How to Build Agents That Improve Over Time: Model, Harness, Context | Interrupt London

## Executive summary

The session outlines a framework for building self-improving AI agents, dividing the agent into three core components: the Model, the Harness, and the Context. Improvement can occur in any of these dimensions, but the common foundation for all improvement loops is robust observability via Traces and Evaluations (Evals). Techniques range from supervised fine-tuning (SFT) for models to automated fix proposals via LangSmith Engine for the harness, culminating in a 'North Star' vision where the agent autonomously detects, evaluates, and fixes its own mistakes.

## Key takeaways

- Agent Architecture (Model, Harness, Context): An agent consists of a Model (the core reasoning engine), a Harness (the system controlling actions, observation, and persistence, e.g., Deep Agents), and Context (external knowledge, files, or tools). Improvement must be planned for all three dimensions.
- Model Improvement Methods: Models can be improved using Supervised Fine-Tuning (SFT) for simpler tasks (classification, extraction) via tools like LangSmith Fine-Tuning (`smithtune` CLI), or using Reinforcement Learning (RL) for more complex tasks, which involves rollouts and real-time scoring.
- The Role of Traces and Evals: Traces (sequences of runs/steps) and Evals are the foundational data required for improvement. Traces are critical for capturing the exact inputs, outputs, and sequence of events. Evals act as the 'new PRD,' defining what 'good' behavior looks like, and are essential for proactive testing.
- Automated Improvement (LangSmith Engine): The LangSmith Engine attempts to automate the improvement of the harness by running over traces, clustering issues, and proposing fixes (e.g., generating a PR, adding tests, or eval data points), though human-in-the-loop review remains critical.

## Technical details

- Agent Observability and Data Structures: A 'run' is a single invocation; multiple runs form a 'trace,' and a conversation forms a 'thread.' Trajectories provide a standardized view over a thread, capturing all necessary inputs, outputs, and event sequences. SmithDB is a purpose-built database for ingesting and querying this new type of agent observability data, addressing semantic and logic failures.
- Evaluation Criteria and Simulation: Defining 'good' in evals is hard. Criteria can judge the expected output (exact match), positive/negative criteria (e.g., 'must mention X'), the end state of the environment (e.g., database changes), or the exact sequence of tool calls. Running these evals requires dedicated simulation environments (mock or dev databases) to prevent hitting production systems.
- Red Teaming for Proactive Error Detection: Red teaming proactively finds agent mistakes. Instead of relying on production failures, techniques include sending two semantically similar queries to check for response consistency, thereby grounding the answer in a known correct reference point.

## Practical implications

- Implement robust observability pipelines (SmithDB) to capture full agent interaction traces, treating traces as a first-class data type.
- Treat agent evaluations (Evals) as a critical part of the Product Requirements Document (PRD), defining success criteria beyond simple output matching.
- Integrate red teaming and consistency checks (e.g., comparing responses to semantically similar prompts) into the testing suite to proactively identify failure modes.
- For build engineers, the improvement loop suggests treating agent logic updates (Harness) as code changes requiring rigorous CI/CD testing against defined eval datasets.

## Topics

Agent Development, LLM Fine-Tuning, Agent Observability, Evaluation Engineering, LangChain, Build Automation, LangSmith Engine, LangSmith Fine-Tuning / smithtune CLI, OpenWiki, Hex, Profound

Source: https://www.youtube.com/watch?v=xy_7SzwsIjM