Reinforcement Learning (RL)

Sep 16, 2026 · 9:36

Where RL Will Take Search — Maximilian-David Rumpf, SID.ai

The presentation outlines how Reinforcement Learning (RL) is poised to revolutionize search by moving beyond traditional, fixed-pipeline architectures. While current agentic search offers vastly higher quality results (roughly doubling the chance of finding correct documents), it is prohibitively expensive and slow (minutes vs. milliseconds). The proposed solution is training a specialized, highly efficient sub-agent using RL, which can adapt its search strategy on the fly, leading to massive improvements in speed and cost compared to frontier models or classical pipelines.

Key takeaways

1. RL Enables Adaptive Search 3:40

   Unlike classical pipelines where decisions are fixed at design time, an RL-trained sub-agent can iterate, search, read results, and refine its query until it is satisfied, making it highly adaptive to complex questions.

2. Significant Performance Gains 8:10

   Training a specialized model using RL results in search quality that is approximately 20 times faster and about 100 times cheaper than using a general frontier model for the same task.

3. Sub-Agents for Efficiency 8:50

   By passing the searching and thinking process to a dedicated, cost-effective sub-agent, the main agent only processes high-quality results, drastically reducing the computational cost associated with context window pollution.

• AI Engineer
• Reinforcement Learning (RL)
• Information Retrieval (Search)
• Agentic AI
• LLM Optimization
• System Architecture
• Maximilian David Rumpf

Watch on YouTube Full article

Sep 13, 2026 · 1:26:47

Build A Reasoning Model Scratch 3: The Verifier for Evaluation and RL with Verifiable Rewards

This video details the implementation of a robust verifier pipeline for evaluating Large Language Models (LLMs) on mathematical benchmarks, specifically using the MATH-500 dataset. The verifier is crucial for establishing a baseline performance metric, which will later be used in Reinforcement Learning with Verifiable Rewards (RLVR) training. The process involves eight key steps: generating text, extracting the final answer (ideally from a boxed format), normalizing the answer to a canonical form, mathematically verifying its equivalence to a ground truth using `sympy`, and finally grading the answer to compute overall model accuracy.

Key takeaways

1. LLM Evaluation Methodologies 2:01

   Model evaluation can be categorized into Multiple Choice, Verifier-based, Leaderboard-based, and LLM Judge methods. The verifier approach is preferred here because it provides an objective evaluation with a hard ground truth answer, which is necessary for RLVR training.

2. The Verifier Pipeline 19:13

   The evaluation pipeline is complex, requiring steps to extract the final answer, normalize the format (e.g., removing LaTeX fluff), verify mathematical equivalence, and grade the result. This robustness is critical for reliable benchmarking.

3. Model Performance Comparison 27:13

   The base model's accuracy (e.g., 15.6% on MATH-500) is significantly lower than the reasoning model's accuracy (e.g., 50.8%). This highlights the value of specialized reasoning techniques and training.

4. Reproducibility Caveats 20:15

   Model evaluation results can vary based on the computing device (CPU, MPS, CUDA) and due to floating-point math, necessitating running evaluations multiple times and averaging the results for robustness.

• Sebastian Raschka
• Large Language Models (LLMs)
• Model Evaluation
• Verifiable Rewards
• Reinforcement Learning (RL)
• Symbolic Mathematics
• Pipeline Engineering
• Reasoning Book
• Reasoning GitHub repo
• Reasoning effort article
• LLMs from Scratch book
• LLMs from Scratch repo
• LLMs from Scratch playlist

Watch on YouTube Full article

Sep 12, 2026 · 51:07

Ask the Experts: Inside Nemotron Post-Training | Nemotron Labs

This session details the advanced post-training pipeline used by NVIDIA AI researchers to build state-of-the-art (SOTA) models like Nemotron, focusing on enhancing intelligence and enabling agentic capabilities. Post-training is presented as an evolution from traditional task-specific training, utilizing structured data, chat templates, and specialized frameworks like NeMo Gym and NeMo RL. Key strategies discussed include performing ablation studies on data subsets, employing Mixture of Experts (MoE) for capability generalization, and establishing robust feedback loops using real-world user data to prevent model degradation.

Key takeaways

1. Post-Training Evolution 3:58

   Post-training builds upon pre-training (which uses massive, diverse, unstructured data for causal language modeling) by focusing on structured data. This teaches the model to follow instructions, use chat templates, and emit tool calls, moving beyond simple task-specific or multi-task training.

2. Structured Data for Capabilities 7:29

   To narrow model focus, post-training emphasizes structured data (e.g., tool responses, user prompts, tool sets) over unstructured text. This allows the model to learn specific formats, such as JSON or XML, for tool interaction.

3. Mitigating Capability Degradation 13:54

   To specialize a model (e.g., for software engineering) without losing general capabilities, techniques include including general chat data in the blend and using algorithmic approaches like multi-teacher on-policy distillation (MOPD).

4. Data Bias and Profiling 19:20

   Data quality is paramount. Researchers must analyze data statistics (e.g., trajectory length, tool call diversity) and perform ablation studies to identify and address biases (e.g., over-reliance on a single tool) that could cause model failure in real-world use.

5. Starting the Pipeline 25:12

   For developers, the process involves defining the model's 'soul' (core capabilities), quantifying these goals via benchmarks, setting quantitative milestones, and iterating through the post-training pipeline. Starting resources include Nemotron 3.5 Lightning.

• NVIDIA Developer
• Large Language Models (LLMs)
• Reinforcement Learning (RL)
• Supervised Fine-Tuning (SFT)
• Data Engineering
• Agentic AI
• Model Optimization
• NeMo Data Designer
• NeMo Gym
• NeMo RL
• Nemotron GitHub
• HuggingFace
• Nemotron 3.5 Lightning

Watch on YouTube Full article

Sep 11, 2026 · 1:13:40

Training Agents 4: From reward functions to environments.

This session details the evolution of agent training from simple functional reward signals to complex, stateful environments. The core concept is the `reset()/step()` contract, which allows agents to interact with a simulated world (the environment) over a sequence of actions. The discussion covers the OpenM framework, which standardizes environment definition, and its integration with TRL (Transformer Reinforcement Learning) using isolated compute environments like Hugging Face sandboxes. This enables training sophisticated agents, such as coding agents (OpenCode), on complex, multi-step tasks while maintaining reproducibility and isolation.

Key takeaways

1. The Shift to Stateful Environments 1:42

   For agents performing sequences of actions (e.g., tool calls, file edits), the reward signal must come from the environment's state after an action, rather than being a simple Python function evaluated once. This requires adopting the standard `reset()/step()` contract, moving from sparse signals to continuous interaction loops.

2. The OpenM Ecosystem 5:30

   OpenM provides a standardized, containerized way to define any task as an environment. It packages the task, the runtime (compute backend), and the grading components (verifiers, rewards) into a single, shareable unit that can be deployed on various platforms (e.g., Hugging Face Spaces, Kubernetes).

3. Advanced Agentic Training Loops 9:10

   Training can be managed by two models: the 'White Box' (where the training framework owns the loop) and the 'Black Box' (where the agent/harness owns the loop). The latter requires a 'capture proxy' to intercept agent calls and send the resulting rollout graph back to the trainer for policy updates.

4. Reproducibility and Isolation 7:30

   Environments are designed to be self-contained applications, often deployed in isolated compute environments (like HF sandboxes). This ensures that training runs are reproducible and prevents the agent from accessing or manipulating the verifiers or task description to 'hack' the reward signal.

• Hugging Face
• Reinforcement Learning (RL)
• Agentic Systems
• Build Engineering
• Containerization
• Hugging Face Ecosystem
• LLM Training
• OpenM
• Hugging Face Hub
• TRL (Transformer Reinforcement Learning)
• OpenCode

Watch on YouTube Full article

Aug 21, 2026 · 34:41

Hugging Face Journal Club: Training AI Scientists to Replicate Research

The discussion summarizes research on Faraday-27B, a model trained by Inherent designed for scientific replication—the ability to reproduce results from redacted ML/AI papers. The system uses Reinforcement Learning (RL) and integrates CodeX as a tool, allowing the agent to execute code within a simulated environment. Key methodological advances include using sophisticated rubric-based judges (generated via Claude) instead of simple verifiers, employing multi-rollout averaging to mitigate variance, and implementing weighted credit assignment across the agent's steps.

Key takeaways

1. Scientific Replication Task

   The model is tasked with replicating missing figures from redacted ML/AI papers. This process requires the agent to use tools (like CodeX) and execute code in a simulated environment, moving toward full automation of AI R&D.

2. Advanced Judging Mechanism 0:01

   Instead of simple verification, the system uses a rubric-based judge (generated by Claude) that assigns fine-grained points for correct reasoning, figure accuracy, and code writing. This process involves averaging judgments across multiple rollouts to prevent reward hacking.

3. Performance & Scaling 0:02

   The trained Faraday model demonstrated strong performance, sometimes outperforming much larger models like Claude and GPT-5. Furthermore, the system showed generalization even when given increased compute resources (e.g., scaling up to 8 hours/8 B300s).

• Hugging Face
• AI Research
• Reinforcement Learning (RL)
• Large Language Models (LLMs)
• Tool Use
• Scientific Computing
• Faraday-27B Model
• Hugging Face Paper Link

Watch on YouTube Full article

Aug 14, 2026 · 17:46

From RL to IRL — Gaurav Mishra, Amazon AGI Lab

The talk details the transition from Reinforcement Learning (RL) in controlled environments ('the game') to real-world deployment (IRL), where agents face significant complexities. While RL is effective for tasks with verifiable outcomes and multiple solution paths, real life introduces partial observability, irreversible actions, expiring credentials, and adversarial content. To bridge this gap, the speaker proposes a 'flight school' approach: training agents in high-fidelity digital sandboxes that simulate messiness (e.g., layout shifts, pop-ups). System improvements include implementing Process Reward Models (penalizing dangerous steps along the path), utilizing Calibrated Confidence (teaching the agent when to escalate to a user), and building robust 'harnesses' with guardrails for checkpointing, rollback, and risk classification.

Key takeaways

1. RL vs. IRL: The Core Challenge 9:07

   RL works well in controlled environments where the outcome is verifiable. However, when deployed in real life (IRL), agents encounter partial observability (e.g., DOM missing content baked into images) and irreversible actions, causing failures like account lockouts or redirection to malicious sites.

2. The 'Flight School' Approach 13:49

   Instead of focusing only on the final outcome (exams), agents must be trained in messy, high-fidelity simulations that model real-world edge cases like slow loads, focus stealing, and random account states. Recovery actions (refresh, backtrack) must become native model capabilities.

3. System Architecture Improvements

   Robust agent systems require upgrading the 'cockpit' (the harness). This includes adding guardrails for action risk classification, implementing checkpointing and rollback mechanisms, and requiring calibrated confidence to determine when human handoff is necessary.

• AI Engineer
• Reinforcement Learning (RL)
• AI Agents
• Partial Observability
• Process Reward Modeling
• Guardrails
• Digital Sandboxes
• Computer Vision/Grounding
• Amazon AGI Lab

Watch on YouTube Full article

Aug 11, 2026 · 33:25

Hugging Face Journal Club: Direct On-Policy Distillation

The discussion details a novel technique called Direct On-Policy Distillation for achieving weak-to-strong generalization in large language models. This method proposes an efficient alternative to expensive full Reinforcement Learning (RL) training by leveraging the policy shift observed when training a small model with RL. Specifically, it uses this policy shift as a dense reward signal to train and update a much larger target model (student), significantly reducing computational costs compared to direct RL on the large model.

Key takeaways

1. Weak-to-Strong Generalization via Policy Shift

   Instead of directly training a large model with expensive RL, this method measures how an RL run changes a small model's policy (the 'policy shift'). This shift is then used as a dense reward signal to distill knowledge into the larger target model.

2. Efficiency Gains

   The technique offers substantial cost savings. For example, training a 7B model via RL might take 320 hours, while using distillation from a 1.5B model's policy shift can reduce the estimated time to around 164 hours.

3. Methodological Blurring

   The process blurs the line between traditional RL and knowledge distillation by combining two types of losses: the policy shift signal (from RL) and a standard KL term, making the overall training setup highly efficient.

• Hugging Face
• Reinforcement Learning (RL)
• Knowledge Distillation
• Large Language Models (LLMs)
• Weak-to-Strong Generalization
• Model Optimization
• Weak-to-Strong Generalization via Direct On-Policy Distillation

Watch on YouTube Full article

Aug 4, 2026 · 30:57

Hugging Face Journal Club: Scaling Laws for Pre-training & RL

The discussion analyzes a paper proposing a joint scaling law for pre-training and Reinforcement Learning (RL), which models how compute allocation across these stages impacts downstream task performance. Key findings suggest that increasing compute allocated to pre-training significantly boosts the model's ability on specific tasks (Pass 1) but has diminishing returns on generalized capability (Pass K). The analysis highlights the importance of optimizing the trade-off between SFT and RL compute budgets.

Key takeaways

1. Pre-training vs. RL Compute Allocation 15:20

   The primary takeaway is that increasing compute allocated to pre-training leads to higher performance on downstream tasks (Pass 1). Conversely, while RL improves Pass 1, the model's generalized capability (Pass K) remains relatively stable regardless of the pre-training scale.

2. Scaling Laws and Model Size 24:45

   When fixing the total compute budget, training smaller models for longer is generally more effective than attempting to train larger models, contradicting simple Chinchilla scaling assumptions in certain contexts.

• Hugging Face
• Scaling Laws
• Large Language Models (LLMs)
• Reinforcement Learning (RL)
• Model Optimization
• Compute Budgeting
• Understanding Reasoning from Pretraining to Post-Training

Watch on YouTube Full article

Aug 3, 2026 · 1:12:59

Stanford CS329A Self-Improving AI Agents | Part 3 | Robust Verification

This lecture traces the evolution of verification methods for Large Language Model (LLM) outputs, aiming to close the generation-verification gap. The discussion covers four major research advancements: training verifiers using outcome-based reward models (GSM8K), implementing process-based supervision via PRMs (PRM800K), automating annotation with Math-Shepherd, and finally, combining multiple weak verifiers into a robust system using Weaver. Key findings highlight that process supervision is generally superior to outcome supervision, and ensembling multiple specialized verifiers significantly boosts model accuracy on complex reasoning tasks.

Key takeaways

1. Process Supervision vs. Outcome Supervision 26:00

   While outcome-based reward models (ORM) only check the final answer's correctness, process-based reward models (PRM) assign rewards per step of reasoning. PRMs are superior because they manage false positives better and encourage interpretable, human-endorsed steps [2:36:00].

2. Ensembling Weak Verifiers (Weaver) 23:02

   The Weaver approach combines multiple weak verifiers (e.g., LLM judges, reward models) using techniques like Naive Bayes or logistic regression to create a single, highly capable verifier. This method significantly improves performance by leveraging the collective signal of diverse sources [3:42:00].

3. Data Efficiency and Scaling 26:00

   PRMs are shown to be more data-efficient than ORMs. Furthermore, the lecture demonstrates that scaling verification by increasing the number of verifiers (rather than just sampling more completions) can improve results while maintaining computational efficiency [3:42:00].

4. The Role of Self-Improvement 17:36

   Advanced techniques involve using the model itself to generate data (e.g., Math-Shepherd) and then training a PRM on this synthetic, semi-automated data, allowing the system to self-improve its reasoning capabilities [2:56:00].

• Stanford Online
• Large Language Models (LLMs)
• Verification
• Reward Modeling
• Reinforcement Learning (RL)
• Ensemble Methods
• Test-Time Scaling
• Reasoning Benchmarks
• GSM8K Dataset
• PRM800K Dataset
• Stanford CS329A Self-Improving AI Agents

Watch on YouTube Full article

Aug 3, 2026 · 1:12:39

Stanford CS329A Self-Improving AI Agents | Part 6 | Train Time Scaling/Scaling RL

This lecture explores advanced techniques for scaling Large Language Model (LLM) reasoning capabilities through 'train-time scaling' and Reinforcement Learning (RL). The discussion covers three key papers: STaR (Self-Taught Reasoner), DeepSeekMath, and DAPO. Key findings demonstrate that smaller models can achieve high accuracy on complex benchmarks like AIME by leveraging structured training data generation (STaR) or advanced RL algorithms (DAPO/GRPO). The core insight is that closing the feedback loop—using model outputs to improve the model itself—is crucial for boosting reasoning, especially in domains with verifiability.

Key takeaways

1. Train-Time Scaling vs. Test-Time Scaling 2:00

   While test-time scaling (inference-based techniques like majority voting) improves accuracy by sampling outputs, train-time scaling uses the model's own filtered outputs to fine-tune and improve the model weights directly, creating a powerful closed feedback loop.

2. STaR Boosts Reasoning via Rationalization 6:30

   The STaR method bootstraps reasoning by generating solutions on a small set of examples. It filters for correct answers and then generates rationales (hints) for incorrect attempts, allowing the model to learn from failed paths iteratively.

3. GRPO Addresses RL Memory Constraints 10:05

   DeepSeekMath introduced Group Relative Policy Optimization (GRPO), an efficient alternative to PPO that reduces memory overhead by using a group baseline instead of maintaining multiple policy copies, enabling scaling RL to larger models.

4. DAPO Stabilizes Complex Reasoning 17:30

   DAPO addresses training instability in long chain-of-thought reasoning by implementing asymmetric clipping (allowing bigger increases) and dynamic sampling (filtering out zero or one reward groups to maintain a useful gradient signal).

• Stanford Online
• Reinforcement Learning (RL)
• Large Language Models (LLMs)
• Reasoning Capabilities
• Self-Improving Agents
• Train-Time Scaling
• Model Optimization
• Stanford CS329A Self-Improving AI Agents
• Agentic AI professional education program

Watch on YouTube Full article

Jul 31, 2026 · 18:20

Learning on the Job: The Future of Post-Training — Raymond Feng, Applied Compute

The presentation outlines a vision for advanced AI agents that can continuously learn and adapt in real-world enterprise environments post-deployment. The core methodology involves an orchestrated training loop: interactions are captured by an orchestrator, processed through inference engines, graded, and the resulting data (graded chats/traces) is fed into a training engine to compute weight updates. Key challenges addressed include environment fidelity, preventing reward hacking, and transitioning from controlled, replayable synthetic environments to uncontrolled, off-policy production data.

Key takeaways

1. The Progression of Agent Training

   Agent training moves from simple single-turn Q&A tasks (controlled by a dedicated training stack) to complex, multi-turn, long-horizon tasks that require offloading environment state outside the training stack.

2. The Core RL Training Loop

   Training relies on an orchestrator driving rollouts, which sends prompts to a model and then passes results to a grader. The resulting graded chats are used by a training engine to compute weight updates for the inference engines.

3. Addressing Real-World Data Challenges 15:45

   Replicating production environments is difficult due to issues like non-replayability and off-policy data. The future requires methods like automated data pipelines and qualitative feedback ingestion to learn from real interactions.

4. The Vision: Self-Improving Agents

   The ultimate goal is a single deployment model that can interact across many different settings, continuously self-evaluating and computing weight updates from every interaction it has.

• AI Engineer
• Reinforcement Learning (RL)
• AI Agents
• Model Training Pipelines
• Data Engineering
• System Orchestration
• Nvidia Polar

Watch on YouTube Full article

Jul 31, 2026 · 19:27

Reinforcement Learning without Verifiable Rewards — Will Brown, Prime Intellect

Will Brown discusses extending Reinforcement Learning (RL) into complex, real-world tasks that lack clean, verifiable rewards. The core thesis is that 'environments' must serve as the anchor for learning. Techniques like grounding in source material, using LLM judges to audit actions, and employing a reverse direction trick are necessary to generate reliable reward signals when ground truth is unavailable. The ultimate goal is enabling continual learning—allowing deployed agents to autonomously improve by observing and correcting mistakes in messy production settings.

Key takeaways

1. The Shift from Verifiable Rewards 6:53

   Traditional RL thrives on verifiable rewards (e.g., math, code test cases). However, most real-world tasks (like writing reports or handling refunds) are fuzzy and lack clean best answers, requiring new methods to generate reliable signals.

2. Environments as the Learning Anchor

   An 'environment' is defined by a task, a harness (e.g., Docker image, codebase), and a scoring rule/verifier. These objects can be used not only for RL but also for Supervised Fine-Tuning (SFT) or prompt optimization.

3. Mitigating Reward Hacking

   Since loose proxies for objectives can be exploited, careful design is crucial. Techniques include inspecting traces, running small experiments, and using judges to audit rollouts in hindsight.

4. Generating Signal via Reverse Direction 17:26

   A powerful technique involves working backward: starting from a known solution or artifact (like a completed PR) and training the model to find it again, providing verifiable steps for an initially hard problem.

• AI Engineer
• Reinforcement Learning (RL)
• AI Agents
• Continual Learning
• System Testing
• LLMs
• Build Engineering
• Will Brown
• Prime Intellect (Primordial AI)

Watch on YouTube Full article

Jul 31, 2026 · 20:14

Agents at Scale: Inside MiniMax's Model and the Infrastructure Behind It — Olive Song

The discussion details the engineering stack and open-source philosophy behind MiniMax's model, M3. Olive Song emphasizes that the open weights approach allows the community to build upon and optimize the model, fostering widespread intelligence access. The technical focus covers advanced training techniques—including multimodality (text, image, video) and Reinforcement Learning (RL) for long-horizon tasks like replicating academic papers (12-hour runs)—and the complex infrastructure required for deployment. Key engineering challenges discussed include writing specialized GPU kernels, optimizing the inference stack from 'day zero,' managing KV cache growth in agentic workflows, and adapting to shifting workloads from chat-based to multi-turn, tool-calling agents.

Key takeaways

1. Open Weights Philosophy 2:07

   MiniMax advocates for open source because it aligns with their mission of making intelligence widely accessible. By releasing weights, they enable developers (like Together AI) to optimize the model's inference speed and capabilities through community contributions.

2. Multimodality Training 8:02

   MiniMax M3 is multimodal, understanding text, code, images, and videos. Crucially, it was trained multimodally from scratch to prevent 'training collapse,' ensuring that the modalities naturally interact (e.g., visual tokens attending to text tokens).

3. Agentic Workloads and Inference Shifts 13:40

   The workload is shifting from simple chat turns to complex agentic workflows involving hundreds of multi-turn tool calls. This requires significant optimization in the inference stack, particularly concerning KV cache management and routing.

4. Long-Horizon RL Tasks

   Training for complex tasks (e.g., replicating an ICLR paper over 12 hours) requires careful formulation of the problem, defining environments, and optimizing reward functions within the Reinforcement Learning framework.

• AI Engineer
• Large Language Models (LLMs)
• Reinforcement Learning (RL)
• GPU Optimization
• Inference Engineering
• Open Source AI
• Multimodality
• MiniMax M3
• OS World
• SVG bench / Kernel bench

Watch on YouTube Full article

Jul 29, 2026 · 36:59

Stanford CS229 Machine Learning | Spring 2026 | Lecture 1: Introduction

This lecture provides a high-level introduction to Machine Learning fundamentals (Supervised, Unsupervised, and Reinforcement Learning) within the context of modern AI. The course emphasizes understanding the mathematical foundations and core techniques behind algorithms rather than focusing on programming implementation. Key topics include model training using large datasets, advanced concepts like embeddings, and the architectural differences between traditional ML tasks and general-purpose Large Language Models (LLMs). A critical focus for system builders is placed on the necessity of optimizing ML systems for hardware compatibility and speed.

Key takeaways

1. ML Paradigm Shift 6:11

   The field has moved from specific, task-oriented models to general-purpose agents (LLMs). While traditional methods still apply, the focus is on tuning fundamental model capabilities rather than building complex data pipelines for every single use case.

2. The Importance of ML Systems 33:00

   A planned lecture will cover 'ML system,' which addresses the critical need to make software and hardware compatible. Optimizing algorithms for speed (e.g., making them run 2x faster) is crucial due to the high cost associated with AI computation.

3. Reinforcement Learning (RL) in LLMs 24:50

   RL can be used to train models dealing with stochastic sampling, such as text generation. Techniques like Policy Gradient and using human feedback (e.g., RLHF/RAG) are necessary because the generation process is not differentiable.

• Stanford Online
• Machine Learning Fundamentals
• Supervised Learning (SFT)
• Unsupervised Learning (Clustering, LSA)
• Reinforcement Learning (RL)
• Large Language Models (LLMs)
• Diffusion Models
• ML System Optimization
• CS229 Machine Learning
• Stanford AI Programs Info

Watch on YouTube Full article

Jul 22, 2026 · 44:21

Stanford CS547 HCI Seminar | Spring 2026 | Promoting Agency in Human-AI Interaction

This seminar explores designing AI systems that promote user agency when LLMs are used as personal advisers (coaches/counselors), rather than mere assistants. The core thesis is that successful health behavior change requires systems to elicit qualitative context and navigate uncertainty to provide non-prescriptive support. Practical implementations, such as the GPT coach chatbot and the Bloom iOS application, demonstrate how integrating motivational interviewing strategies with wearable data can improve user mindset and sense of control. Algorithmically, the work proposes 'zero-shot Bayesian Adaptive Planning' using LLMs to strategically balance asking informative questions versus acting on known information by modeling latent uncertainty over the user's state.

Key takeaways

1. Shift from Assistant to Adviser 2:00

   LLM usage is shifting toward deeply personal advice (e.g., health, relationships), requiring a design paradigm that augments the user rather than automating tasks. This necessitates non-prescriptive support.

2. The Role of Qualitative Context 4:00

   Effective coaching relies on eliciting qualitative context (goals, values, motivations) over quantitative data (step count, heart rate). This aligns with principles from Motivational Interviewing.

3. GPT coach Implementation 8:10

   The GPT coach chatbot uses three prompt chains—Dialogue State Chain, Motivational Interviewing Chain, and Tool Use Chain—to integrate qualitative coaching strategies with quantitative data from the Apple Health Kit API.

4. Algorithmic Solution: Bayesian Adaptive Planning 17:55

   To improve strategic decision-making, the proposed algorithm uses Reinforcement Learning (RL) theory to model latent uncertainty ($ heta$) over the user's state. The agent must learn to strategically trade off asking informative questions against acting on current knowledge.

• Stanford Online
• Human-AI Interaction
• Behavior Change Theory
• Large Language Models (LLMs)
• Reinforcement Learning (RL)
• Bayesian Inference
• Health Technology
• Apple Health Kit API
• Stanford HCI Seminar Schedule

Watch on YouTube Full article

Jul 21, 2026 · 30:27

Hugging Face Journal Club: AsyncOPD and How Stale Can On-Policy Distillation Be?

The discussion details Asynchronous On-Policy Distillation (AsyncOPD), a method designed to significantly boost training throughput by making the distillation process fully asynchronous. While conventional methods are synchronous and suffer from GPU blocking during backpropagation, AsyncOPD continuously generates rollouts from policies while simultaneously scoring them with a teacher model. This approach achieves substantial speedups (1.5x to 2.7x) compared to synchronous methods, though it introduces complexity related to maintaining stability when student rollouts become significantly off-policy.

Key takeaways

1. AsyncOPD for Throughput Gains 23:23

   By decoupling the generator (student policy), scorer (teacher model), and backpropagator, AsyncOPD eliminates GPU blocking inherent in synchronous distillation. This allows continuous operation, leading to throughput improvements of 1.5x to 2.7x on various math benchmarks.

2. Addressing Cache Misses via Monte Carlo Sampling 30:07

   When calculating Reverse KL divergence using Top-K logits, cache misses can occur because the required log probabilities for the loss calculation may not have been stored during the initial sampling phase. MC sampling is proposed as a solution to estimate the loss accurately by storing and correcting estimates using important sampling.

3. Trade-offs in Off-Policy Distillation 20:50

   While fully asynchronous methods offer high throughput, they require careful handling of off-policy rollouts. The stability and accuracy are dependent on the degree of staleness allowed (the difference between the current policy and the teacher's distribution).

• Hugging Face
• On-Policy Distillation
• Asynchronous Training
• Reinforcement Learning (RL)
• KL Divergence
• Monte Carlo Sampling
• Throughput Optimization
• Paper: 2606.24143

Watch on YouTube Full article

Jul 16, 2026 · 1:41:04

🔬 RL with Verifiable Rewards, but the Verifier is a Lab — Lila Sciences

Lila Sciences proposes that the next frontier of data generation—the 'next internet-scale dataset'—will come from running the scientific method as a closed-loop Reinforcement Learning (RL) process. The wet lab acts as the verifier and data source for training general AI models. This approach aims to create an 'infinite token generator' by synthesizing knowledge across biology, chemistry, and materials science into a single reasoning model, which is then offered via a scalable 'zero-FTE startup' platform.

Key takeaways

1. The Lab as Data Center 20:30

   The future scientific facility must function like a data center, prioritizing dense packing and energy efficiency. The infrastructure uses planar motor systems and a physical transport layer (analogized to a PCI bus) to connect instruments for seamless, automated operation.

2. Scientific Superintelligence via RL 40:50

   The core thesis is that science can be an 'infinite token generator.' By using the scientific method and nature as verifiers in a closed-loop system, models generate verifiable reasoning tokens (e.g., 10 trillion tokens across multiple domains) that improve general intelligence, proving that 'breadth gives us depth.'

3. The Zero-FTE Startup Model 1:18:20

   Lila Sciences commercializes its platform by allowing external partners to run entire scientific programs (e.g., CAR-T development or novel material synthesis) over a short period using the model and automated lab infrastructure, without needing to build their own physical facility.

• Latent Space
• AI for Science
• Reinforcement Learning (RL)
• Materials Science
• Computational Chemistry
• Lab Automation
• Generative AI
• Escalante Bio: Your Experiment has a Runtime
• Lila Sciences

Watch on YouTube Full article

Jul 16, 2026 · 36:00

New Model: Inkling by Thinking Machine on Hugging Face

Thinking Machines announced Inkling, a massive open model with nearly one trillion parameters. It is designed as a natively multimodal architecture, accepting image, text, and audio inputs simultaneously. Key technical features include a Mixture-of-Experts (MoE) structure (975B total / 41B active params), a 1M token context window, and advanced deployment support across multiple frameworks like `transformers`, SGLang, vLLM, and `llama.cpp`. The model is available in BF16 and NVFP4 formats, facilitating high-performance inference on various hardware setups.

Key takeaways

1. Multimodal Capability & Scale 3:50

   Inkling is a true multimodal model that natively processes image, text, and audio inputs using a single architecture. It boasts an immense 1M token context window and was trained on 45T tokens.

2. Architectural Innovations 5:10

   The model utilizes relative attention (replacing RoPE) and a specialized SConv layer to efficiently aggregate hidden states, making it highly efficient for multimodal tasks compared to previous models.

3. Deployment Flexibility 7:10

   Inkling supports multiple deployment paths: BF16 (requiring ~2TB VRAM) and NVFP4 (600GB VRAM). It provides day-zero support for `transformers`, SGLang, vLLM, and `llama.cpp` (including GGUF quants via Unsloth), enabling diverse inference environments.

• Hugging Face
• Large Language Models
• Multimodality
• Mixture-of-Experts (MoE)
• Quantization
• Inference Optimization
• Reinforcement Learning (RL)
• Thinking Machines Inkling Model (BF16)
• Thinking Machines Inkling Model (NVFP4)
• GGUF Quants (Unsloth)
• Full Blog Post

Watch on YouTube Full article