# Improving Agents is a Data Mining Problem — Vivek Trivedy, LangChain

## Executive summary

The continuous improvement of autonomous agents requires shifting focus from code determinism to data mining agent traces. The speaker argues that observability and continual learning are fundamentally linked: an agent's actions in an environment generate a trace record that serves as the substrate for all future improvements. Techniques like harness engineering, distillation (SFT), and analyzing counterfactual traces allow developers to systematically improve agents at lower costs than relying solely on frontier models.

## Key takeaways

- Shipping is the First Step: To gather data for improvement, an agent must be deployed into a real-world environment (shipping it). This process generates valuable trace data from tool calls, API usage, and CLIs.
- Observability = Continual Learning: There is a tight coupling between observability and continual learning for agents. Both require comprehensive traces—the record of actions taken in the environment—to allow the agent to update its internal knowledge or definition.
- The Value of Traces: Traces capture fine-grained behavior that simple pass/fail benchmarks miss. They are crucial for proving counterfactuals (e.g., comparing GPT 5.5 vs. GLM 5.2) and understanding how agents behave at a granular level.
- Improvement Loop Strategy: For agent improvement, the recommended loop is: Start with Harness Engineering (fast feedback, ~2 minutes) $\rightarrow$ Saturate this ceiling $\rightarrow$ Fine-tune the model to break through it $\rightarrow$ Return to Harness Engineering.
- The Future of Data: Agent activity will generate data at an exponential rate, potentially eclipsing all human-produced data in history. Managing this requires building systems that can efficiently mine and process traces.

## Technical details

- Agent Traces: Traces are the comprehensive records of an agent's operation, including tool calls, API interactions, and output messages. They serve as the primary data source for understanding complex autonomous behavior.
- Harness Engineering: A method of systematic testing that provides instant feedback on agent performance using defined evaluation sets (evals). It is recommended as the initial, rapid improvement step for agents.
- Distillation and Fine-Tuning: Using high-performing frontier models (e.g., Opus) to generate 'good traces' or examples, which are then used in a dataset to fine-tune smaller, cheaper open models (e.g., 9B or 13B models). This process aims to mimic complex behavior at lower cost.
- Feedback Signals: Agents benefit most from dense feedback signals, rather than binary pass/fail metrics. Traces provide this fine-grained signal, allowing agents to pinpoint exactly where they failed.
- Memory Management: For long-term agent operation (years/decades), simple append-only logs are insufficient. Agents require sophisticated memory systems that can efficiently update and query state over time.

## Practical implications

- Implement robust tracing and observability tools immediately upon deploying any agent to capture full operational context.
- Prioritize using trace data for systematic improvement loops (Harness $\rightarrow$ Fine-tuning $\rightarrow$ Harness) rather than relying solely on prompt engineering.
- When evaluating models, use traces to perform counterfactual analysis, comparing different model versions or architectures against the same task.
- For high-volume inference workloads, consider shifting economic focus from token costs (input/output tokens) to dedicated hardware cluster costs.

## Topics

Agentic Systems, Observability, Continual Learning, Data Mining, LLM Optimization, Harness Engineering, Model Distillation, LangChain, Harvey Legal Benchmark

Source: https://www.youtube.com/watch?v=CvRngaQZQ3Y