# From RL to IRL — Gaurav Mishra, Amazon AGI Lab

## Executive summary

The talk details the transition from Reinforcement Learning (RL) in controlled environments ('the game') to real-world deployment (IRL), where agents face significant complexities. While RL is effective for tasks with verifiable outcomes and multiple solution paths, real life introduces partial observability, irreversible actions, expiring credentials, and adversarial content. To bridge this gap, the speaker proposes a 'flight school' approach: training agents in high-fidelity digital sandboxes that simulate messiness (e.g., layout shifts, pop-ups). System improvements include implementing Process Reward Models (penalizing dangerous steps along the path), utilizing Calibrated Confidence (teaching the agent when to escalate to a user), and building robust 'harnesses' with guardrails for checkpointing, rollback, and risk classification.

## Key takeaways

- RL vs. IRL: The Core Challenge: RL works well in controlled environments where the outcome is verifiable. However, when deployed in real life (IRL), agents encounter partial observability (e.g., DOM missing content baked into images) and irreversible actions, causing failures like account lockouts or redirection to malicious sites.
- The 'Flight School' Approach: Instead of focusing only on the final outcome (exams), agents must be trained in messy, high-fidelity simulations that model real-world edge cases like slow loads, focus stealing, and random account states. Recovery actions (refresh, backtrack) must become native model capabilities.
- System Architecture Improvements: Robust agent systems require upgrading the 'cockpit' (the harness). This includes adding guardrails for action risk classification, implementing checkpointing and rollback mechanisms, and requiring calibrated confidence to determine when human handoff is necessary.

## Technical details

- RL Mechanics: In RL, the agent's policy receives a reward computed over an entire generation, differing from SFT/pre-training which assigns loss per token. Algorithms mentioned include PPO and GRPO.
- Agent Capabilities: Coding agents can be extended to general computer use tasks (e.g., reading emails, filing receipts) because these actions can be represented as code via APIs or tools like Playwright JavaScript Web MCP.
- Process Reward Model: This model focuses on penalizing dangerous steps throughout the entire trajectory, rather than only scoring the final outcome. It emphasizes path safety over mere success.
- System Guardrails (The Cockpit): The harness must include: Checkpointing and rollback for risky states; Action risk classifier to vet proposed actions; Credential guardrails to detect session expiry; Execution monitor to catch unproductive behavior.

## Practical implications

- AI agent development requires shifting testing focus from ideal success paths to failure mode simulation (e.g., network flakiness, UI changes).
- Build engineers must integrate 'risk-aware execution' into CI/CD pipelines for agents, treating actions as potentially irreversible.
- The concept of a robust 'harness' or guardrail layer is critical for production deployment, acting as an external safety net around the core model logic.

## Topics

Reinforcement Learning (RL), AI Agents, Partial Observability, Process Reward Modeling, Guardrails, Digital Sandboxes, Computer Vision/Grounding, Amazon AGI Lab

Source: https://www.youtube.com/watch?v=Cc0_nyxROBA