# Brian Douglas - The beginners guide to training AI on your own code - AI Native DevCon June 2026

## Executive summary

This talk outlines a closed-loop system for training AI agents using proprietary code execution traces. The core concept involves capturing every agent interaction (telemetry) via an open-source proxy (tapes.dev), structuring this data into a content-addressable Merkle DAG. This raw trace data is then processed by streaming tools like Kafka and Flink SQL to detect anomalies, which feeds back into the agent's next run—creating a 'self-healing loop.' The captured traces are used for supervised fine-tuning (SFT) or Direct Preference Optimization (DPO) to embed learned skills directly into local models, eliminating reliance on external training data.

## Key takeaways

- Closed-Loop Agent Training: Agents generate data by running in an environment; this telemetry is captured and used immediately for model improvement. This creates a self-healing infrastructure loop, allowing agents to learn from failures (anomalies) and successes.
- Data Capture Infrastructure: The open-source proxy, tapes.dev, intercepts LLM API calls to build a content-addressable Merkle DAG of every conversation turn, requiring zero instrumentation.
- Anomaly Detection and Feedback: Streaming pipelines (Kafka/Flink SQL) run continuous anomaly detection on session data (e.g., stuck loops, token spikes). Alerts generated by Flink are read by the agent before its next execution, enabling self-correction.
- Knowledge Transfer Methods: Captured traces can be used for Specialized Fine-Tuning (SFT) or Direct Preference Optimization (DPO). SFT embeds skills into the model, while DPO uses preference data derived from successful vs. failed sessions.

## Technical details

- Data Structure and Capture: Tapes.dev captures agent sessions into a Merkle DAG (Directed Acyclic Graph), ensuring every conversation turn is content-addressable and filterable, even across multiple parallel agents.
- Streaming Pipeline: The pipeline involves publishing events to Kafka, where Flink SQL runs continuous anomaly detection queries (e.g., stuck loop detection in tumbling windows) before generating alerts for the agent.
- Model Fine-Tuning: The process involves using raw telemetry to generate clean training datasets, which are then used to fine-tune local models. Specialized Fine-Tuning (SFT) is described as adding 'notes' or skills into the model's knowledge base.
- Computational Constraints: The speaker notes that while advanced techniques like DPO are possible, they are computationally expensive (e.g., requiring specialized hardware like H100s) and may not be practical for general use.

## Practical implications

- Build engineers can apply the concept of continuous feedback loops to CI/CD pipelines by treating failed deployments or test runs as valuable 'telemetry' that informs model retraining or system hardening.
- The Merkle DAG structure provides a robust, verifiable audit trail for all development decisions and agent interactions, improving traceability in complex systems.
- By implementing local data capture (like tapes.dev), teams can reduce dependency on expensive third-party API calls and maintain full control over their proprietary knowledge base.

## Topics

AI Agent Development, Machine Learning Operations (MLOps), Data Telemetry, Continuous Integration/Deployment (CI/CD), tapes.dev, Kafka / Flink SQL, Pi Game Boy

Source: https://www.youtube.com/watch?v=o-IunU6b1t8