Learning on the Job: The Future of Post-Training — Raymond Feng, Applied Compute
The presentation outlines a vision for advanced AI agents that can continuously learn and adapt in real-world enterprise environments post-deployment. The core methodology involves an orchestrated training loop: interactions are captured by an orchestrator, processed through inference engines, graded, and the resulting data (graded chats/traces) is fed into a training engine to compute weight updates. Key challenges addressed include environment fidelity, preventing reward hacking, and transitioning from controlled, replayable synthetic environments to uncontrolled, off-policy production data.
Key takeaways
-
The Progression of Agent Training
Agent training moves from simple single-turn Q&A tasks (controlled by a dedicated training stack) to complex, multi-turn, long-horizon tasks that require offloading environment state outside the training stack.
-
The Core RL Training Loop
Training relies on an orchestrator driving rollouts, which sends prompts to a model and then passes results to a grader. The resulting graded chats are used by a training engine to compute weight updates for the inference engines.
-
Addressing Real-World Data Challenges
15:45
Replicating production environments is difficult due to issues like non-replayability and off-policy data. The future requires methods like automated data pipelines and qualitative feedback ingestion to learn from real interactions.
-
The Vision: Self-Improving Agents
The ultimate goal is a single deployment model that can interact across many different settings, continuously self-evaluating and computing weight updates from every interaction it has.