The Video Signal

Topic

Model Quantization

All digests tagged Model Quantization

Taking Reinforcement Learning Cross Datacenter — Nan Jiang, Modal thumbnail

· 19:50

Taking Reinforcement Learning Cross Datacenter — Nan Jiang, Modal

This talk addresses scaling Reinforcement Learning (RL) post-training across geographically distributed compute resources by fundamentally changing the synchronization unit. Instead of shipping massive full checkpoints (up to 500 GB), the proposed method uses a small 'lossless patch' representing only the changes in visible weights, shrinking the transfer size from hundreds of gigabytes to potentially hundreds of megabytes. This enables the rollout fleet to operate elastically across different regions and providers, decoupling it from the central training cluster.

Key takeaways

  1. Decoupling RL Training from Compute Location 5:22

    The standard RL loop couples the trainer and rollout worker in a single fast-fabric cluster. The solution proposes that the 'rollout serving island'—a coherent endpoint or local group of endpoints—is the movable unit, allowing it to operate across scattered, autoscaled capacity (the 'bazaar') rather than being restricted to one perfect cluster (the 'cathedral').

  2. Sparse Weight Updates via Adam Absorption 8:05

    The core mechanism relies on the fact that while gradients are dense, the actual change in the served weight view is extremely small. This 'Adam absorption' phenomenon occurs because a typical Adam step (around 3 millionths) is far smaller than the BF16 rounding boundary (around 0.0039), meaning the visible value does not change significantly.

  3. Lossless Patch Synchronization 9:50

    The synchronization unit is redefined as a 'lossless patch' (a diff) rather than a full checkpoint. This patch, which includes the change index and replacement bits, allows the rollout engine to bitwise reconstruct the exact served version from a much smaller object.

Watch on YouTube Full article

Hugging Face Journal Club: Kimi K3 thumbnail

· 41:28

Hugging Face Journal Club: Kimi K3

The discussion summarizes the Kimi K3 tech report, detailing a highly complex and scaled training pipeline for large language models. Key innovations include specialized agentic structures (general, deep research, coding), multi-tier on-policy distillation across nine domain experts, and advanced infrastructure techniques like partial rollout schemes and dynamic resource scheduling. The talk emphasizes that the model's performance is achieved through sophisticated engineering efforts rather than a single breakthrough concept.

Key takeaways

  1. Multi-Tier Expert Specialization 2:00

    The training pipeline involves generating nine specialized domain experts (3 reasoning levels: low, high, max) for three general domains (general tasks, deep research, coding). These are combined using multi-tier on-policy distillation into a single student model.

  2. Partial Rollout Scheme 4:08

    A novel RL technique where the system samples $k$ rollouts from $n$ prompts. It uses an additional parameter $\lambda$ to measure completed rollouts within a budget, updating weights on that subset and pushing incomplete rollouts into subsequent iterations.

  3. Extensible Chat Template 6:42

    The Kimi K3 model utilizes an 'extensible token markup language' for its chat template. This design aims to be easy to learn during SFT while remaining flexible enough to accommodate future modalities without re-engineering the core template.

  4. Advanced Infrastructure Scaling 30:35

    The system employs sophisticated resource management, including a dynamic rollout auto throttling scheduler based on KV cache pressure and a method for collocating RL training and inference by automatically balancing GPU resources between the two tasks.

Watch on YouTube Full article