# Reinforcement Learning without Verifiable Rewards — Will Brown, Prime Intellect

## Executive summary

Will Brown discusses extending Reinforcement Learning (RL) into complex, real-world tasks that lack clean, verifiable rewards. The core thesis is that 'environments' must serve as the anchor for learning. Techniques like grounding in source material, using LLM judges to audit actions, and employing a reverse direction trick are necessary to generate reliable reward signals when ground truth is unavailable. The ultimate goal is enabling continual learning—allowing deployed agents to autonomously improve by observing and correcting mistakes in messy production settings.

## Key takeaways

- The Shift from Verifiable Rewards: Traditional RL thrives on verifiable rewards (e.g., math, code test cases). However, most real-world tasks (like writing reports or handling refunds) are fuzzy and lack clean best answers, requiring new methods to generate reliable signals.
- Environments as the Learning Anchor: An 'environment' is defined by a task, a harness (e.g., Docker image, codebase), and a scoring rule/verifier. These objects can be used not only for RL but also for Supervised Fine-Tuning (SFT) or prompt optimization.
- Mitigating Reward Hacking: Since loose proxies for objectives can be exploited, careful design is crucial. Techniques include inspecting traces, running small experiments, and using judges to audit rollouts in hindsight.
- Generating Signal via Reverse Direction: A powerful technique involves working backward: starting from a known solution or artifact (like a completed PR) and training the model to find it again, providing verifiable steps for an initially hard problem.

## Technical details

- RL Framework: The process involves an agent (model) interacting with a task/world environment in a loop. RL updates the policy weights by calculating an advantage (Reward - Baseline), aiming to maximize the gradient toward higher rewards.
- Grounding and Supervision: Grounding uses source material (e.g., doc corpora, production traces) to provide context-based supervision. An A/B test comparing performance with and without this context can exploit a 'capability gap' to generate learning signal.
- Judges and Search: LLMs are used as 'judges' to spend compute making decisions on whether an action was good or bad. Scaling search involves using these judges to extract detailed rubrics from failed rollouts, which then guide future training.
- Continual Learning Goal: The ultimate goal is for models to be deployed in complex settings and learn autonomously. This requires making the process of environment design, monitoring, and refinement highly automated.

## Practical implications

- For build engineers managing complex systems, this framework suggests treating the entire operational environment (including CI pipelines and tool use) as a trainable 'environment' for agents.
- The concept of using production traces as source material allows teams to continuously refine model behavior and detect failure modes without needing perfect upfront supervision or labeled data.
- Implementing automated 'judges' that audit agent rollouts can significantly improve the quality of testing beyond simple unit tests, especially in multi-step workflows.

## Topics

Reinforcement Learning (RL), AI Agents, Continual Learning, System Testing, LLMs, Build Engineering, Will Brown, Prime Intellect (Primordial AI)

Source: https://www.youtube.com/watch?v=AQv3qRCG6Gw