From RL to IRL — Gaurav Mishra, Amazon AGI Lab
The talk details the transition from Reinforcement Learning (RL) in controlled environments ('the game') to real-world deployment (IRL), where agents face significant complexities. While RL is effective for tasks with verifiable outcomes and multiple solution paths, real life introduces partial observability, irreversible actions, expiring credentials, and adversarial content. To bridge this gap, the speaker proposes a 'flight school' approach: training agents in high-fidelity digital sandboxes that simulate messiness (e.g., layout shifts, pop-ups). System improvements include implementing Process Reward Models (penalizing dangerous steps along the path), utilizing Calibrated Confidence (teaching the agent when to escalate to a user), and building robust 'harnesses' with guardrails for checkpointing, rollback, and risk classification.
Key takeaways
-
RL vs. IRL: The Core Challenge
9:07
RL works well in controlled environments where the outcome is verifiable. However, when deployed in real life (IRL), agents encounter partial observability (e.g., DOM missing content baked into images) and irreversible actions, causing failures like account lockouts or redirection to malicious sites.
-
The 'Flight School' Approach
13:49
Instead of focusing only on the final outcome (exams), agents must be trained in messy, high-fidelity simulations that model real-world edge cases like slow loads, focus stealing, and random account states. Recovery actions (refresh, backtrack) must become native model capabilities.
-
System Architecture Improvements
Robust agent systems require upgrading the 'cockpit' (the harness). This includes adding guardrails for action risk classification, implementing checkpointing and rollback mechanisms, and requiring calibrated confidence to determine when human handoff is necessary.
