Improving Agents is a Data Mining Problem — Vivek Trivedy, LangChain
The continuous improvement of autonomous agents requires shifting focus from code determinism to data mining agent traces. The speaker argues that observability and continual learning are fundamentally linked: an agent's actions in an environment generate a trace record that serves as the substrate for all future improvements. Techniques like harness engineering, distillation (SFT), and analyzing counterfactual traces allow developers to systematically improve agents at lower costs than relying solely on frontier models.
Key takeaways
-
Shipping is the First Step
To gather data for improvement, an agent must be deployed into a real-world environment (shipping it). This process generates valuable trace data from tool calls, API usage, and CLIs.
-
Observability = Continual Learning
4:04
There is a tight coupling between observability and continual learning for agents. Both require comprehensive traces—the record of actions taken in the environment—to allow the agent to update its internal knowledge or definition.
-
The Value of Traces
6:00
Traces capture fine-grained behavior that simple pass/fail benchmarks miss. They are crucial for proving counterfactuals (e.g., comparing GPT 5.5 vs. GLM 5.2) and understanding how agents behave at a granular level.
-
Improvement Loop Strategy
13:00
For agent improvement, the recommended loop is: Start with Harness Engineering (fast feedback, ~2 minutes) $\rightarrow$ Saturate this ceiling $\rightarrow$ Fine-tune the model to break through it $\rightarrow$ Return to Harness Engineering.
-
The Future of Data
17:00
Agent activity will generate data at an exponential rate, potentially eclipsing all human-produced data in history. Managing this requires building systems that can efficiently mine and process traces.