Brian Douglas - The beginners guide to training AI on your own code - AI Native DevCon June 2026
This talk outlines a closed-loop system for training AI agents using proprietary code execution traces. The core concept involves capturing every agent interaction (telemetry) via an open-source proxy (tapes.dev), structuring this data into a content-addressable Merkle DAG. This raw trace data is then processed by streaming tools like Kafka and Flink SQL to detect anomalies, which feeds back into the agent's next run—creating a 'self-healing loop.' The captured traces are used for supervised fine-tuning (SFT) or Direct Preference Optimization (DPO) to embed learned skills directly into local models, eliminating reliance on external training data.
Key takeaways
-
Closed-Loop Agent Training
3:50
Agents generate data by running in an environment; this telemetry is captured and used immediately for model improvement. This creates a self-healing infrastructure loop, allowing agents to learn from failures (anomalies) and successes.
-
Data Capture Infrastructure
10:30
The open-source proxy, tapes.dev, intercepts LLM API calls to build a content-addressable Merkle DAG of every conversation turn, requiring zero instrumentation.
-
Anomaly Detection and Feedback
2:00
Streaming pipelines (Kafka/Flink SQL) run continuous anomaly detection on session data (e.g., stuck loops, token spikes). Alerts generated by Flink are read by the agent before its next execution, enabling self-correction.
-
Knowledge Transfer Methods
22:00
Captured traces can be used for Specialized Fine-Tuning (SFT) or Direct Preference Optimization (DPO). SFT embeds skills into the model, while DPO uses preference data derived from successful vs. failed sessions.
