Reinforcement Learning (RL)

Jul 31, 2026 · 20:14

Agents at Scale: Inside MiniMax's Model and the Infrastructure Behind It — Olive Song

The discussion details the engineering stack and open-source philosophy behind MiniMax's model, M3. Olive Song emphasizes that the open weights approach allows the community to build upon and optimize the model, fostering widespread intelligence access. The technical focus covers advanced training techniques—including multimodality (text, image, video) and Reinforcement Learning (RL) for long-horizon tasks like replicating academic papers (12-hour runs)—and the complex infrastructure required for deployment. Key engineering challenges discussed include writing specialized GPU kernels, optimizing the inference stack from 'day zero,' managing KV cache growth in agentic workflows, and adapting to shifting workloads from chat-based to multi-turn, tool-calling agents.

Key takeaways

1. Open Weights Philosophy 2:07

   MiniMax advocates for open source because it aligns with their mission of making intelligence widely accessible. By releasing weights, they enable developers (like Together AI) to optimize the model's inference speed and capabilities through community contributions.

2. Multimodality Training 8:02

   MiniMax M3 is multimodal, understanding text, code, images, and videos. Crucially, it was trained multimodally from scratch to prevent 'training collapse,' ensuring that the modalities naturally interact (e.g., visual tokens attending to text tokens).

3. Agentic Workloads and Inference Shifts 13:40

   The workload is shifting from simple chat turns to complex agentic workflows involving hundreds of multi-turn tool calls. This requires significant optimization in the inference stack, particularly concerning KV cache management and routing.

4. Long-Horizon RL Tasks

   Training for complex tasks (e.g., replicating an ICLR paper over 12 hours) requires careful formulation of the problem, defining environments, and optimizing reward functions within the Reinforcement Learning framework.

• AI Engineer
• Large Language Models (LLMs)
• Reinforcement Learning (RL)
• GPU Optimization
• Inference Engineering
• Open Source AI
• Multimodality
• MiniMax M3
• OS World
• SVG bench / Kernel bench

Watch on YouTube Full article

Jul 29, 2026 · 36:59

Stanford CS229 Machine Learning | Spring 2026 | Lecture 1: Introduction

This lecture provides a high-level introduction to Machine Learning fundamentals (Supervised, Unsupervised, and Reinforcement Learning) within the context of modern AI. The course emphasizes understanding the mathematical foundations and core techniques behind algorithms rather than focusing on programming implementation. Key topics include model training using large datasets, advanced concepts like embeddings, and the architectural differences between traditional ML tasks and general-purpose Large Language Models (LLMs). A critical focus for system builders is placed on the necessity of optimizing ML systems for hardware compatibility and speed.

Key takeaways

1. ML Paradigm Shift 6:11

   The field has moved from specific, task-oriented models to general-purpose agents (LLMs). While traditional methods still apply, the focus is on tuning fundamental model capabilities rather than building complex data pipelines for every single use case.

2. The Importance of ML Systems 33:00

   A planned lecture will cover 'ML system,' which addresses the critical need to make software and hardware compatible. Optimizing algorithms for speed (e.g., making them run 2x faster) is crucial due to the high cost associated with AI computation.

3. Reinforcement Learning (RL) in LLMs 24:50

   RL can be used to train models dealing with stochastic sampling, such as text generation. Techniques like Policy Gradient and using human feedback (e.g., RLHF/RAG) are necessary because the generation process is not differentiable.

• Stanford Online
• Machine Learning Fundamentals
• Supervised Learning (SFT)
• Unsupervised Learning (Clustering, LSA)
• Reinforcement Learning (RL)
• Large Language Models (LLMs)
• Diffusion Models
• ML System Optimization
• CS229 Machine Learning
• Stanford AI Programs Info

Watch on YouTube Full article

Jul 22, 2026 · 44:21

Stanford CS547 HCI Seminar | Spring 2026 | Promoting Agency in Human-AI Interaction

This seminar explores designing AI systems that promote user agency when LLMs are used as personal advisers (coaches/counselors), rather than mere assistants. The core thesis is that successful health behavior change requires systems to elicit qualitative context and navigate uncertainty to provide non-prescriptive support. Practical implementations, such as the GPT coach chatbot and the Bloom iOS application, demonstrate how integrating motivational interviewing strategies with wearable data can improve user mindset and sense of control. Algorithmically, the work proposes 'zero-shot Bayesian Adaptive Planning' using LLMs to strategically balance asking informative questions versus acting on known information by modeling latent uncertainty over the user's state.

Key takeaways

1. Shift from Assistant to Adviser 2:00

   LLM usage is shifting toward deeply personal advice (e.g., health, relationships), requiring a design paradigm that augments the user rather than automating tasks. This necessitates non-prescriptive support.

2. The Role of Qualitative Context 4:00

   Effective coaching relies on eliciting qualitative context (goals, values, motivations) over quantitative data (step count, heart rate). This aligns with principles from Motivational Interviewing.

3. GPT coach Implementation 8:10

   The GPT coach chatbot uses three prompt chains—Dialogue State Chain, Motivational Interviewing Chain, and Tool Use Chain—to integrate qualitative coaching strategies with quantitative data from the Apple Health Kit API.

4. Algorithmic Solution: Bayesian Adaptive Planning 17:55

   To improve strategic decision-making, the proposed algorithm uses Reinforcement Learning (RL) theory to model latent uncertainty ($ heta$) over the user's state. The agent must learn to strategically trade off asking informative questions against acting on current knowledge.

• Stanford Online
• Human-AI Interaction
• Behavior Change Theory
• Large Language Models (LLMs)
• Reinforcement Learning (RL)
• Bayesian Inference
• Health Technology
• Apple Health Kit API
• Stanford HCI Seminar Schedule

Watch on YouTube Full article

Jul 21, 2026 · 30:27

Hugging Face Journal Club: AsyncOPD and How Stale Can On-Policy Distillation Be?

The discussion details Asynchronous On-Policy Distillation (AsyncOPD), a method designed to significantly boost training throughput by making the distillation process fully asynchronous. While conventional methods are synchronous and suffer from GPU blocking during backpropagation, AsyncOPD continuously generates rollouts from policies while simultaneously scoring them with a teacher model. This approach achieves substantial speedups (1.5x to 2.7x) compared to synchronous methods, though it introduces complexity related to maintaining stability when student rollouts become significantly off-policy.

Key takeaways

1. AsyncOPD for Throughput Gains 23:23

   By decoupling the generator (student policy), scorer (teacher model), and backpropagator, AsyncOPD eliminates GPU blocking inherent in synchronous distillation. This allows continuous operation, leading to throughput improvements of 1.5x to 2.7x on various math benchmarks.

2. Addressing Cache Misses via Monte Carlo Sampling 30:07

   When calculating Reverse KL divergence using Top-K logits, cache misses can occur because the required log probabilities for the loss calculation may not have been stored during the initial sampling phase. MC sampling is proposed as a solution to estimate the loss accurately by storing and correcting estimates using important sampling.

3. Trade-offs in Off-Policy Distillation 20:50

   While fully asynchronous methods offer high throughput, they require careful handling of off-policy rollouts. The stability and accuracy are dependent on the degree of staleness allowed (the difference between the current policy and the teacher's distribution).

• Hugging Face
• On-Policy Distillation
• Asynchronous Training
• Reinforcement Learning (RL)
• KL Divergence
• Monte Carlo Sampling
• Throughput Optimization
• Paper: 2606.24143

Watch on YouTube Full article

Jul 16, 2026 · 1:41:04

🔬 RL with Verifiable Rewards, but the Verifier is a Lab — Lila Sciences

Lila Sciences proposes that the next frontier of data generation—the 'next internet-scale dataset'—will come from running the scientific method as a closed-loop Reinforcement Learning (RL) process. The wet lab acts as the verifier and data source for training general AI models. This approach aims to create an 'infinite token generator' by synthesizing knowledge across biology, chemistry, and materials science into a single reasoning model, which is then offered via a scalable 'zero-FTE startup' platform.

Key takeaways

1. The Lab as Data Center 20:30

   The future scientific facility must function like a data center, prioritizing dense packing and energy efficiency. The infrastructure uses planar motor systems and a physical transport layer (analogized to a PCI bus) to connect instruments for seamless, automated operation.

2. Scientific Superintelligence via RL 40:50

   The core thesis is that science can be an 'infinite token generator.' By using the scientific method and nature as verifiers in a closed-loop system, models generate verifiable reasoning tokens (e.g., 10 trillion tokens across multiple domains) that improve general intelligence, proving that 'breadth gives us depth.'

3. The Zero-FTE Startup Model 1:18:20

   Lila Sciences commercializes its platform by allowing external partners to run entire scientific programs (e.g., CAR-T development or novel material synthesis) over a short period using the model and automated lab infrastructure, without needing to build their own physical facility.

• Latent Space
• AI for Science
• Reinforcement Learning (RL)
• Materials Science
• Computational Chemistry
• Lab Automation
• Generative AI
• Escalante Bio: Your Experiment has a Runtime
• Lila Sciences

Watch on YouTube Full article

Jul 16, 2026 · 36:00

New Model: Inkling by Thinking Machine on Hugging Face

Thinking Machines announced Inkling, a massive open model with nearly one trillion parameters. It is designed as a natively multimodal architecture, accepting image, text, and audio inputs simultaneously. Key technical features include a Mixture-of-Experts (MoE) structure (975B total / 41B active params), a 1M token context window, and advanced deployment support across multiple frameworks like `transformers`, SGLang, vLLM, and `llama.cpp`. The model is available in BF16 and NVFP4 formats, facilitating high-performance inference on various hardware setups.

Key takeaways

1. Multimodal Capability & Scale 3:50

   Inkling is a true multimodal model that natively processes image, text, and audio inputs using a single architecture. It boasts an immense 1M token context window and was trained on 45T tokens.

2. Architectural Innovations 5:10

   The model utilizes relative attention (replacing RoPE) and a specialized SConv layer to efficiently aggregate hidden states, making it highly efficient for multimodal tasks compared to previous models.

3. Deployment Flexibility 7:10

   Inkling supports multiple deployment paths: BF16 (requiring ~2TB VRAM) and NVFP4 (600GB VRAM). It provides day-zero support for `transformers`, SGLang, vLLM, and `llama.cpp` (including GGUF quants via Unsloth), enabling diverse inference environments.

• Hugging Face
• Large Language Models
• Multimodality
• Mixture-of-Experts (MoE)
• Quantization
• Inference Optimization
• Reinforcement Learning (RL)
• Thinking Machines Inkling Model (BF16)
• Thinking Machines Inkling Model (NVFP4)
• GGUF Quants (Unsloth)
• Full Blog Post

Watch on YouTube Full article