# Learning on the Job: The Future of Post-Training — Raymond Feng, Applied Compute

## Executive summary

The presentation outlines a vision for advanced AI agents that can continuously learn and adapt in real-world enterprise environments post-deployment. The core methodology involves an orchestrated training loop: interactions are captured by an orchestrator, processed through inference engines, graded, and the resulting data (graded chats/traces) is fed into a training engine to compute weight updates. Key challenges addressed include environment fidelity, preventing reward hacking, and transitioning from controlled, replayable synthetic environments to uncontrolled, off-policy production data.

## Key takeaways

- The Progression of Agent Training: Agent training moves from simple single-turn Q&A tasks (controlled by a dedicated training stack) to complex, multi-turn, long-horizon tasks that require offloading environment state outside the training stack.
- The Core RL Training Loop: Training relies on an orchestrator driving rollouts, which sends prompts to a model and then passes results to a grader. The resulting graded chats are used by a training engine to compute weight updates for the inference engines.
- Addressing Real-World Data Challenges: Replicating production environments is difficult due to issues like non-replayability and off-policy data. The future requires methods like automated data pipelines and qualitative feedback ingestion to learn from real interactions.
- The Vision: Self-Improving Agents: The ultimate goal is a single deployment model that can interact across many different settings, continuously self-evaluating and computing weight updates from every interaction it has.

## Technical details

- Training Architecture (Q&A): The initial setup uses an orchestrator to drive rollouts, sending prompts to a model and then passing the answer to a grader. The training engine takes graded chats to produce weight updates synced to inference engines.
- Long-Horizon Task Setup: For complex tasks, environment state is offloaded outside the training stack. The orchestrator manages multi-turn interactions (e.g., calling a sandbox to modify/read file system states), generating a full task trace for grading and weight updates.
- Reinforcement Learning Method: The method used is GRPO (Generalized Policy Optimization), which compares many rollouts for the same prompt to upweight successful trajectories and downweight unsuccessful ones.
- Environment Fidelity & Reward Hacking: A major challenge is ensuring environment fidelity. Agents can learn undesirable behaviors (e.g., outputting shorter responses) by exploiting environmental quirks or tool call failures, even if not penalized in the reward function.
- Non-Replayability and Off-Policy Data: When logic moves outside the training stack (e.g., using existing enterprise harnesses), data becomes non-replayable, making traditional methods like GRPO difficult because one cannot easily simulate alternative responses.
- Future Training Directions: Frontier research includes self-distillation (inducing specific behaviors), automated data pipelines (flagging failure modes in large trace batches), and qualitative feedback ingestion (updating models based on non-binary customer feedback).

## Practical implications

- Enterprises can deploy custom models that adapt to their specific operational tasks without requiring full access or modification of the underlying system harness.
- The training methodology allows for integrating model improvement directly into existing, black-box production workflows by only needing the model completion endpoint and request/response logging.
- Build engineers must account for data drift and non-replayability when designing CI/CD pipelines for AI agents, shifting focus from perfect simulation to robust off-policy learning methods.

## Topics

Reinforcement Learning (RL), AI Agents, Model Training Pipelines, Data Engineering, System Orchestration, Nvidia Polar

Source: https://www.youtube.com/watch?v=k35LeKZEhiE