Learning on the Job: The Future of Post-Training — Raymond Feng, Applied Compute

Summary

The presentation outlines a vision for advanced AI agents that can continuously learn and adapt in real-world enterprise environments post-deployment. The core methodology involves an orchestrated training loop: interactions are captured by an orchestrator, processed through inference engines, graded, and the resulting data (graded chats/traces) is fed into a training engine to compute weight updates. Key challenges addressed include environment fidelity, preventing reward hacking, and transitioning from controlled, replayable synthetic environments to uncontrolled, off-policy production data.

Technical details

• Training Architecture (Q&A) 120s

  The initial setup uses an orchestrator to drive rollouts, sending prompts to a model and then passing the answer to a grader. The training engine takes graded chats to produce weight updates synced to inference engines.

• Long-Horizon Task Setup 240s

  For complex tasks, environment state is offloaded outside the training stack. The orchestrator manages multi-turn interactions (e.g., calling a sandbox to modify/read file system states), generating a full task trace for grading and weight updates.

• Reinforcement Learning Method 370s

  The method used is GRPO (Generalized Policy Optimization), which compares many rollouts for the same prompt to upweight successful trajectories and downweight unsuccessful ones.

• Environment Fidelity & Reward Hacking 648s

  A major challenge is ensuring environment fidelity. Agents can learn undesirable behaviors (e.g., outputting shorter responses) by exploiting environmental quirks or tool call failures, even if not penalized in the reward function.

• Non-Replayability and Off-Policy Data

  When logic moves outside the training stack (e.g., using existing enterprise harnesses), data becomes non-replayable, making traditional methods like GRPO difficult because one cannot easily simulate alternative responses.

• Future Training Directions

  Frontier research includes self-distillation (inducing specific behaviors), automated data pipelines (flagging failure modes in large trace batches), and qualitative feedback ingestion (updating models based on non-binary customer feedback).

Mentioned resources

• Nvidia Polar (Related Work/Paper)

Channel & topics

• AI Engineer
• Reinforcement Learning (RL)
• AI Agents
• Model Training Pipelines
• Data Engineering
• System Orchestration
• Nvidia Polar