From RL to IRL — Gaurav Mishra, Amazon AGI Lab
Summary
The talk details the transition from Reinforcement Learning (RL) in controlled environments ('the game') to real-world deployment (IRL), where agents face significant complexities. While RL is effective for tasks with verifiable outcomes and multiple solution paths, real life introduces partial observability, irreversible actions, expiring credentials, and adversarial content. To bridge this gap, the speaker proposes a 'flight school' approach: training agents in high-fidelity digital sandboxes that simulate messiness (e.g., layout shifts, pop-ups). System improvements include implementing Process Reward Models (penalizing dangerous steps along the path), utilizing Calibrated Confidence (teaching the agent when to escalate to a user), and building robust 'harnesses' with guardrails for checkpointing, rollback, and risk classification.
Key takeaways
-
RL vs. IRL: The Core Challenge
9:07
RL works well in controlled environments where the outcome is verifiable. However, when deployed in real life (IRL), agents encounter partial observability (e.g., DOM missing content baked into images) and irreversible actions, causing failures like account lockouts or redirection to malicious sites.
-
The 'Flight School' Approach
13:49
Instead of focusing only on the final outcome (exams), agents must be trained in messy, high-fidelity simulations that model real-world edge cases like slow loads, focus stealing, and random account states. Recovery actions (refresh, backtrack) must become native model capabilities.
-
System Architecture Improvements
Robust agent systems require upgrading the 'cockpit' (the harness). This includes adding guardrails for action risk classification, implementing checkpointing and rollback mechanisms, and requiring calibrated confidence to determine when human handoff is necessary.
Technical details
-
RL Mechanics
0s
In RL, the agent's policy receives a reward computed over an entire generation, differing from SFT/pre-training which assigns loss per token. Algorithms mentioned include PPO and GRPO.
-
Agent Capabilities
326s
Coding agents can be extended to general computer use tasks (e.g., reading emails, filing receipts) because these actions can be represented as code via APIs or tools like Playwright JavaScript Web MCP.
-
Process Reward Model
954s
This model focuses on penalizing dangerous steps throughout the entire trajectory, rather than only scoring the final outcome. It emphasizes path safety over mere success.
-
System Guardrails (The Cockpit)
The harness must include: Checkpointing and rollback for risky states; Action risk classifier to vet proposed actions; Credential guardrails to detect session expiry; Execution monitor to catch unproductive behavior.
Mentioned resources
- Amazon AGI Lab
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.