How to Build Agents That Improve Over Time: Model, Harness, Context | Interrupt London
Summary
The session outlines a framework for building self-improving AI agents, dividing the agent into three core components: the Model, the Harness, and the Context. Improvement can occur in any of these dimensions, but the common foundation for all improvement loops is robust observability via Traces and Evaluations (Evals). Techniques range from supervised fine-tuning (SFT) for models to automated fix proposals via LangSmith Engine for the harness, culminating in a 'North Star' vision where the agent autonomously detects, evaluates, and fixes its own mistakes.
Key takeaways
-
Agent Architecture (Model, Harness, Context)
An agent consists of a Model (the core reasoning engine), a Harness (the system controlling actions, observation, and persistence, e.g., Deep Agents), and Context (external knowledge, files, or tools). Improvement must be planned for all three dimensions.
-
Model Improvement Methods
5:22
Models can be improved using Supervised Fine-Tuning (SFT) for simpler tasks (classification, extraction) via tools like LangSmith Fine-Tuning (`smithtune` CLI), or using Reinforcement Learning (RL) for more complex tasks, which involves rollouts and real-time scoring.
-
The Role of Traces and Evals
15:11
Traces (sequences of runs/steps) and Evals are the foundational data required for improvement. Traces are critical for capturing the exact inputs, outputs, and sequence of events. Evals act as the 'new PRD,' defining what 'good' behavior looks like, and are essential for proactive testing.
-
Automated Improvement (LangSmith Engine)
9:12
The LangSmith Engine attempts to automate the improvement of the harness by running over traces, clustering issues, and proposing fixes (e.g., generating a PR, adding tests, or eval data points), though human-in-the-loop review remains critical.
Technical details
-
Agent Observability and Data Structures
1044s
A 'run' is a single invocation; multiple runs form a 'trace,' and a conversation forms a 'thread.' Trajectories provide a standardized view over a thread, capturing all necessary inputs, outputs, and event sequences. SmithDB is a purpose-built database for ingesting and querying this new type of agent observability data, addressing semantic and logic failures.
-
Evaluation Criteria and Simulation
1202s
Defining 'good' in evals is hard. Criteria can judge the expected output (exact match), positive/negative criteria (e.g., 'must mention X'), the end state of the environment (e.g., database changes), or the exact sequence of tool calls. Running these evals requires dedicated simulation environments (mock or dev databases) to prevent hitting production systems.
-
Red Teaming for Proactive Error Detection
Red teaming proactively finds agent mistakes. Instead of relying on production failures, techniques include sending two semantically similar queries to check for response consistency, thereby grounding the answer in a known correct reference point.
Mentioned resources
- LangSmith Engine
- LangSmith Fine-Tuning / smithtune CLI
- OpenWiki
- Hex
- Profound
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.