Reinforcement Learning

While Large Language Models (LLMs) have shown stunning ability in text-based reasoning, they are fundamentally limited to text descriptions of the world and cannot support physical planning or causality. The goal is to build embodied AGI that operates in the physical world.

Training physical AI agents traditionally requires thousands of hours of costly teleoperation. The solution is building accurate simulations that allow for effective transfer to the real world, enabling scalable training for robotics and industrial automation.

Moonlake AI reconstructs a world model from a partial observation (photo/video) by separating the background from manipulable objects. It uses web research (RAG style) to fill in missing information (e.g., the contents of a closed tea box) and models objects with physical properties.

The system relies on neuro-symbolic representations, using code as a powerful substrate. This allows the model to be controllable, editable, and maintainable by incorporating physics engines and symbolic logic, moving beyond purely pixel-based generation.

The progression moves from conventional LLMs to reasoning models, which are modified versions of regular LLMs. These reasoning models form the 'engine' used by modern agent harnesses (e.g., OpenAIs Code Agent) [1:16].

Implementing models from scratch provides unambiguous, precise code examples that are highly valuable for deep learning understanding, serving as a 'proof' beyond mere conceptual images [8:23].

The recommended setup involves cloning the GitHub repository and using `uv` (a fast dependency manager) to sync dependencies within an isolated virtual environment before running code in JupyterLab or VS Code [20:40].

Before fine-tuning a model, thoroughly audit the evaluation setup. Improvements can often be found by adjusting sampling parameters (e.g., using temperature > 0), swapping harnesses (like Pi for open-source control), or increasing resource allocation/timeouts. [0:08:19]

Every evaluation consists of three parts: the Task Set (data, prompts, tools); the Harness (the program loop driving LLM interaction with an environment); and the Scoring Function/Reward Function (which can be deterministic or use a judge LLM). [0:00:55]

RL is a powerful, advanced training algorithm used to improve model capabilities on specific tasks by learning from trial and error. It should be considered the last step after optimizing the eval environment. [0:13:24]

RL is highly sensitive to weak or poorly designed evals. Models may learn to optimize for a proxy metric (reward hacking) rather than solving the true underlying task, necessitating careful evaluation design and red teaming. [0:22:24]

Policy optimization methods (like TRPO and PPO) define a surrogate objective function to estimate the policy gradient, enabling continuous updates. PPO uses a clipped ratio ($ ext{clip}(r_{ heta}, 1- ext{eps}, 1+ ext{eps})$) to constrain the new policy's divergence from the old one, stabilizing training without requiring complex second-order optimization.

The basic model-based recipe (collect data $ ightarrow$ fit dynamics $P(s'|s, a)$ $ ightarrow$ plan) fails when dealing with complex or nonlinear dynamics because extrapolation outside the observed state distribution is unreliable. This issue of generalization and distribution shift must be addressed.

To improve model-based planning, uncertainty quantification is necessary. The distinction between Aleatoric (inherent noise) and Epistemic (model uncertainty) is crucial. Bayesian approaches treat this by modeling the posterior distribution over parameters ($ heta$), allowing for prediction averaging across all plausible models.

A practical approach to estimate model uncertainty is using ensembles: training multiple independent neural network copies. The average of their predictions approximates the predictive posterior distribution, effectively exploring multimodal solution landscapes without requiring complex analytical derivations.

Monte Carlo methods estimate the expected return ($G_t$) by rolling out an episode until a terminal state, requiring full episodes. Temporal Difference (TD) learning improves upon this by using bootstrapping—defining the target as the instantaneous reward plus the discounted future value ($ ext{Reward} + ext{Discounted Future Value}$), allowing for online updates and handling non-terminal environments.

SARSA is an on-policy algorithm, meaning it improves the policy ($ ext{e.g., } ext{epsilon-greedy}$) that is actively used to generate data in the environment. Q-learning is off-policy; it learns about a target optimal policy (the greedy policy) while using data generated by a different behavior policy (also $ ext{epsilon-greedy}$), which is crucial for utilizing historical or simulated data.

To overcome the curse of dimensionality inherent in tabular value function representations, RL methods transition to parametric functions (e.g., neural networks) that approximate $V(s)$ or $Q(s, a)$. This allows generalization across states and controls.

Deep Q Networks (DQN) stabilize learning using two methods: Experience Replay (storing transitions in a buffer to decorrelate samples, satisfying the IID assumption required for regression) and Fixed Q Targets (using a delayed copy of the network parameters ($ ext{Q}_{ ext{target}}$) to prevent the target from being a moving variable during optimization).

The standard RL loop couples the trainer and rollout worker in a single fast-fabric cluster. The solution proposes that the 'rollout serving island'—a coherent endpoint or local group of endpoints—is the movable unit, allowing it to operate across scattered, autoscaled capacity (the 'bazaar') rather than being restricted to one perfect cluster (the 'cathedral').

The core mechanism relies on the fact that while gradients are dense, the actual change in the served weight view is extremely small. This 'Adam absorption' phenomenon occurs because a typical Adam step (around 3 millionths) is far smaller than the BF16 rounding boundary (around 0.0039), meaning the visible value does not change significantly.

The synchronization unit is redefined as a 'lossless patch' (a diff) rather than a full checkpoint. This patch, which includes the change index and replacement bits, allows the rollout engine to bitwise reconstruct the exact served version from a much smaller object.

Teaching AI to hack should follow a ladder structure: from triggering a crash to achieving arbitrary read/write in memory, and ultimately full arbitrary code execution (10:30). This structured approach allows for measurable progress.

Current benchmarks often assume only one vulnerability or use LLMs as judges, which is flawed. The model will tend to 'reward hack' by repeatedly finding the easiest known bug (7:46).

To accurately measure capability, environments must use deterministic graders that check for specific bugs. The proposed 'audit task' allows scoring precision and recall across multiple known and unknown vulnerabilities (14:49).

Testing on the JavaScript engine V8 in Chrome showed that while models achieved high rates of simple crashes, only advanced models could achieve out-of-sandbox exploits (full control flow hijack), demonstrating a clear capability gap (21:10).

RLHF trains LLMs to optimize for human preference (engagement), which makes them excellent assistants but poor autonomous agents. The goal is to please the user, not necessarily to execute a task correctly in a background server environment [7:31].

The core divide in modern AI is between 'assistance' (where the human remains in the loop) and 'automation' (where the system operates autonomously with real stakes). Current models are optimized for the former, making them unreliable for critical business decisions [3:14].

Future AI must shift its optimization target from human preference to verifiable rewards and calibrated decision-making. This requires redesigning the entire AI stack for reliability, moving beyond current LLM post-training methods like RLHF or even RLVR [15:43].

True automation requires smarter software that is more expressive than current SaaS models. The focus must shift from merely automating the writing of code to solving extremely rote, simple tasks that can be done repeatedly and reliably by a computer [10:35].

The policy gradient estimator is necessary because the dependency on parameters ($ heta$) is complex. The fundamental property that $ abla_{ heta} ext{E}_{ ext{P}_{ heta}}[ abla_{ heta} ext{log } ext{P}_{ heta}(a|s)]$ equals zero shows that without a reward function, there are no preferences to optimize for.

PPO is designed to stabilize RL training by using importance sampling and clipping the objective function. This prevents the new policy ($ heta$) from deviating too far from the old policy ($ heta_{old}$), which helps maintain stable learning.

The LLM generation process is modeled as a Markov Decision Process (MDP). The state ($s_t$) includes the history, and the action ($a_t$) is the next generated token. The reward function is typically applied only at the end of the trajectory based on whether the final answer matches the ground truth.

To train models for complex reasoning, RL can be used to reward the entire trajectory based on the final answer's correctness. This approach bypasses the need for explicit labeling of the internal 'thinking tokens,' focusing only on verifiable outcomes.

RL addresses sequential decision-making where actions have long-term ramifications. It requires balancing the trade-off between exploitation (using current best knowledge) and exploration (gathering information). Learning relies on maximizing a scalar reward signal rather than explicit labels or supervision.

An MDP formally describes an environment using five components: State Set ($S$), Action Set ($A$), Transition Dynamics ($P(s'|s, a)$), Reward Function ($R$), and Discount Factor ($\gamma$). The Markov property ensures that the future state transition depends only on the current state and action, not on history.

The value function $V^{\pi}(s)$ estimates the expected total payoff starting at state $s$ under policy $\pi$. The optimal value, $V^*(s)$, is the maximum possible return. These values are solved recursively using the Bellman equation, which relates the current state's value to the expected discounted future rewards.

The Policy Gradient algorithm optimizes a stochastic policy $\pi_{\theta}(a|s)$ by maximizing the expected return $E[R]$. The gradient is computed using the log-probability trick, allowing the calculation of $\nabla_{\theta} E[R]$ through sampling, even when the dependency on $\theta$ only affects the sampling distribution.

Solving major human challenges (e.g., curing cancer) requires patience and thinking in long timeframes, necessitating a fundamental shift in how AI systems are designed for sustained coherence.

The breakthrough that made LLMs usable was not the base model size, but the application of Reinforcement Learning from Human Feedback (RLHF), which provided necessary alignment and structure.

To handle long trajectories, value models (critics) are essential for reducing gradient variance and facilitating credit assignment, which is necessary when rewards are sparse.

The failure of frontier models in real-world tasks (like trading football matches) demonstrated that scaling requires better, more complex environments and simulations, not just larger context windows.

Historically, models like GPT-3 relied heavily on raw web scrapes (e.g., Common Crawl/WebText-2), which constituted up to 85% of the training data. Modern recipes show a significant decrease in web text's proportion, indicating that its value is diminishing relative to code and structured reasoning abilities.

The base model's role is changing from accumulating general world knowledge (the 'prior') to carrying the necessary prior for complex Reinforcement Learning (RL) tasks. RL is no longer a mere 'cherry on top,' but a core component that requires the base model to be prepared for advanced composition and reasoning.

A key trend involves pulling post-training data (like SFT/Q&A chat data) and large-scale synthetic data back into the pre-training phase. This allows models to learn task representations, conversation shapes, and atomic skills from the very beginning.

Gemini Robotics 2 enables models to understand the entire robot's position in space and reason about complex, multi-step tasks (e.g., cleaning a garage), moving beyond simple object manipulation.

The models significantly improve dexterity, allowing robots to perform intricate daily tasks such as folding laundry or precisely unscrewing objects using high-DOF hands.

A new capability allows the robot intelligence to understand when and how to call other robots to accelerate tasks or perform actions in parallel.

The Embodied Reasoning (ER) model will be available via AI Studio and the Gemini Enterprise Agents Platform. An on-device version is also available through a trusted tester program.

The training pipeline involves generating nine specialized domain experts (3 reasoning levels: low, high, max) for three general domains (general tasks, deep research, coding). These are combined using multi-tier on-policy distillation into a single student model.

A novel RL technique where the system samples $k$ rollouts from $n$ prompts. It uses an additional parameter $\lambda$ to measure completed rollouts within a budget, updating weights on that subset and pushing incomplete rollouts into subsequent iterations.

The Kimi K3 model utilizes an 'extensible token markup language' for its chat template. This design aims to be easy to learn during SFT while remaining flexible enough to accommodate future modalities without re-engineering the core template.

The system employs sophisticated resource management, including a dynamic rollout auto throttling scheduler based on KV cache pressure and a method for collocating RL training and inference by automatically balancing GPU resources between the two tasks.

GRPO calculates advantages relative to the group average (Reward - Group Average / Group Standard Deviation). This method eliminates the need for a separate value model, reward model, or critic, simplifying the RL loop. The process involves generating multiple rollouts, scoring them with a verifiable Python function (the reward function), and updating the policy based on these relative advantages.

The training pipeline typically progresses from Supervised Fine-Tuning (SFT) for dense, off-policy signals, to Distillation for richer online rollouts, and finally to RL/GRPO for sparse, on-policy learning. SFT is often used first to bootstrap the model's understanding of the task structure.

Defining a verifiable reward function is critical; it acts as a 'contract' defining success. Rewards can be composed of multiple components, such as a format check (e.g., ensuring JSON structure) and an accuracy check (e.g., passing unit tests). For agentic tasks, using test suites or compiling code provides robust signals.

Monitoring training curves via tools like Track.io is essential for debugging. Key metrics include the reward (should rise), entropy (should remain stable/flat, not dive or spike), and completion length. Failure modes—such as 'reward hacking' (high reward but low test accuracy) or 'collapse' (low entropy)—require deep data inspection.