Reinforcement Learning

Teaching AI to hack should follow a ladder structure: from triggering a crash to achieving arbitrary read/write in memory, and ultimately full arbitrary code execution (10:30). This structured approach allows for measurable progress.

Current benchmarks often assume only one vulnerability or use LLMs as judges, which is flawed. The model will tend to 'reward hack' by repeatedly finding the easiest known bug (7:46).

To accurately measure capability, environments must use deterministic graders that check for specific bugs. The proposed 'audit task' allows scoring precision and recall across multiple known and unknown vulnerabilities (14:49).

Testing on the JavaScript engine V8 in Chrome showed that while models achieved high rates of simple crashes, only advanced models could achieve out-of-sandbox exploits (full control flow hijack), demonstrating a clear capability gap (21:10).

RLHF trains LLMs to optimize for human preference (engagement), which makes them excellent assistants but poor autonomous agents. The goal is to please the user, not necessarily to execute a task correctly in a background server environment [7:31].

The core divide in modern AI is between 'assistance' (where the human remains in the loop) and 'automation' (where the system operates autonomously with real stakes). Current models are optimized for the former, making them unreliable for critical business decisions [3:14].

Future AI must shift its optimization target from human preference to verifiable rewards and calibrated decision-making. This requires redesigning the entire AI stack for reliability, moving beyond current LLM post-training methods like RLHF or even RLVR [15:43].

True automation requires smarter software that is more expressive than current SaaS models. The focus must shift from merely automating the writing of code to solving extremely rote, simple tasks that can be done repeatedly and reliably by a computer [10:35].

The policy gradient estimator is necessary because the dependency on parameters ($ heta$) is complex. The fundamental property that $ abla_{ heta} ext{E}_{ ext{P}_{ heta}}[ abla_{ heta} ext{log } ext{P}_{ heta}(a|s)]$ equals zero shows that without a reward function, there are no preferences to optimize for.

PPO is designed to stabilize RL training by using importance sampling and clipping the objective function. This prevents the new policy ($ heta$) from deviating too far from the old policy ($ heta_{old}$), which helps maintain stable learning.

The LLM generation process is modeled as a Markov Decision Process (MDP). The state ($s_t$) includes the history, and the action ($a_t$) is the next generated token. The reward function is typically applied only at the end of the trajectory based on whether the final answer matches the ground truth.

To train models for complex reasoning, RL can be used to reward the entire trajectory based on the final answer's correctness. This approach bypasses the need for explicit labeling of the internal 'thinking tokens,' focusing only on verifiable outcomes.

RL addresses sequential decision-making where actions have long-term ramifications. It requires balancing the trade-off between exploitation (using current best knowledge) and exploration (gathering information). Learning relies on maximizing a scalar reward signal rather than explicit labels or supervision.

An MDP formally describes an environment using five components: State Set ($S$), Action Set ($A$), Transition Dynamics ($P(s'|s, a)$), Reward Function ($R$), and Discount Factor ($\gamma$). The Markov property ensures that the future state transition depends only on the current state and action, not on history.

The value function $V^{\pi}(s)$ estimates the expected total payoff starting at state $s$ under policy $\pi$. The optimal value, $V^*(s)$, is the maximum possible return. These values are solved recursively using the Bellman equation, which relates the current state's value to the expected discounted future rewards.

The Policy Gradient algorithm optimizes a stochastic policy $\pi_{\theta}(a|s)$ by maximizing the expected return $E[R]$. The gradient is computed using the log-probability trick, allowing the calculation of $\nabla_{\theta} E[R]$ through sampling, even when the dependency on $\theta$ only affects the sampling distribution.

Solving major human challenges (e.g., curing cancer) requires patience and thinking in long timeframes, necessitating a fundamental shift in how AI systems are designed for sustained coherence.

The breakthrough that made LLMs usable was not the base model size, but the application of Reinforcement Learning from Human Feedback (RLHF), which provided necessary alignment and structure.

To handle long trajectories, value models (critics) are essential for reducing gradient variance and facilitating credit assignment, which is necessary when rewards are sparse.

The failure of frontier models in real-world tasks (like trading football matches) demonstrated that scaling requires better, more complex environments and simulations, not just larger context windows.

Historically, models like GPT-3 relied heavily on raw web scrapes (e.g., Common Crawl/WebText-2), which constituted up to 85% of the training data. Modern recipes show a significant decrease in web text's proportion, indicating that its value is diminishing relative to code and structured reasoning abilities.

The base model's role is changing from accumulating general world knowledge (the 'prior') to carrying the necessary prior for complex Reinforcement Learning (RL) tasks. RL is no longer a mere 'cherry on top,' but a core component that requires the base model to be prepared for advanced composition and reasoning.

A key trend involves pulling post-training data (like SFT/Q&A chat data) and large-scale synthetic data back into the pre-training phase. This allows models to learn task representations, conversation shapes, and atomic skills from the very beginning.

Gemini Robotics 2 enables models to understand the entire robot's position in space and reason about complex, multi-step tasks (e.g., cleaning a garage), moving beyond simple object manipulation.

The models significantly improve dexterity, allowing robots to perform intricate daily tasks such as folding laundry or precisely unscrewing objects using high-DOF hands.

A new capability allows the robot intelligence to understand when and how to call other robots to accelerate tasks or perform actions in parallel.

The Embodied Reasoning (ER) model will be available via AI Studio and the Gemini Enterprise Agents Platform. An on-device version is also available through a trusted tester program.

The training pipeline involves generating nine specialized domain experts (3 reasoning levels: low, high, max) for three general domains (general tasks, deep research, coding). These are combined using multi-tier on-policy distillation into a single student model.

A novel RL technique where the system samples $k$ rollouts from $n$ prompts. It uses an additional parameter $\lambda$ to measure completed rollouts within a budget, updating weights on that subset and pushing incomplete rollouts into subsequent iterations.

The Kimi K3 model utilizes an 'extensible token markup language' for its chat template. This design aims to be easy to learn during SFT while remaining flexible enough to accommodate future modalities without re-engineering the core template.

The system employs sophisticated resource management, including a dynamic rollout auto throttling scheduler based on KV cache pressure and a method for collocating RL training and inference by automatically balancing GPU resources between the two tasks.

GRPO calculates advantages relative to the group average (Reward - Group Average / Group Standard Deviation). This method eliminates the need for a separate value model, reward model, or critic, simplifying the RL loop. The process involves generating multiple rollouts, scoring them with a verifiable Python function (the reward function), and updating the policy based on these relative advantages.

The training pipeline typically progresses from Supervised Fine-Tuning (SFT) for dense, off-policy signals, to Distillation for richer online rollouts, and finally to RL/GRPO for sparse, on-policy learning. SFT is often used first to bootstrap the model's understanding of the task structure.

Defining a verifiable reward function is critical; it acts as a 'contract' defining success. Rewards can be composed of multiple components, such as a format check (e.g., ensuring JSON structure) and an accuracy check (e.g., passing unit tests). For agentic tasks, using test suites or compiling code provides robust signals.

Monitoring training curves via tools like Track.io is essential for debugging. Key metrics include the reward (should rise), entropy (should remain stable/flat, not dive or spike), and completion length. Failure modes—such as 'reward hacking' (high reward but low test accuracy) or 'collapse' (low entropy)—require deep data inspection.