Improving Agents is a Data Mining Problem — Vivek Trivedy, LangChain
Summary
The continuous improvement of autonomous agents requires shifting focus from code determinism to data mining agent traces. The speaker argues that observability and continual learning are fundamentally linked: an agent's actions in an environment generate a trace record that serves as the substrate for all future improvements. Techniques like harness engineering, distillation (SFT), and analyzing counterfactual traces allow developers to systematically improve agents at lower costs than relying solely on frontier models.
Key takeaways
-
Shipping is the First Step
To gather data for improvement, an agent must be deployed into a real-world environment (shipping it). This process generates valuable trace data from tool calls, API usage, and CLIs.
-
Observability = Continual Learning
4:04
There is a tight coupling between observability and continual learning for agents. Both require comprehensive traces—the record of actions taken in the environment—to allow the agent to update its internal knowledge or definition.
-
The Value of Traces
6:00
Traces capture fine-grained behavior that simple pass/fail benchmarks miss. They are crucial for proving counterfactuals (e.g., comparing GPT 5.5 vs. GLM 5.2) and understanding how agents behave at a granular level.
-
Improvement Loop Strategy
13:00
For agent improvement, the recommended loop is: Start with Harness Engineering (fast feedback, ~2 minutes) $\rightarrow$ Saturate this ceiling $\rightarrow$ Fine-tune the model to break through it $\rightarrow$ Return to Harness Engineering.
-
The Future of Data
17:00
Agent activity will generate data at an exponential rate, potentially eclipsing all human-produced data in history. Managing this requires building systems that can efficiently mine and process traces.
Technical details
-
Agent Traces
128s
Traces are the comprehensive records of an agent's operation, including tool calls, API interactions, and output messages. They serve as the primary data source for understanding complex autonomous behavior.
-
Harness Engineering
780s
A method of systematic testing that provides instant feedback on agent performance using defined evaluation sets (evals). It is recommended as the initial, rapid improvement step for agents.
-
Distillation and Fine-Tuning
1080s
Using high-performing frontier models (e.g., Opus) to generate 'good traces' or examples, which are then used in a dataset to fine-tune smaller, cheaper open models (e.g., 9B or 13B models). This process aims to mimic complex behavior at lower cost.
-
Feedback Signals
Agents benefit most from dense feedback signals, rather than binary pass/fail metrics. Traces provide this fine-grained signal, allowing agents to pinpoint exactly where they failed.
-
Memory Management
For long-term agent operation (years/decades), simple append-only logs are insufficient. Agents require sophisticated memory systems that can efficiently update and query state over time.
Mentioned resources
- LangChain
- Harvey Legal Benchmark
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.