Taking Reinforcement Learning Cross Datacenter — Nan Jiang, Modal
Summary
This talk addresses scaling Reinforcement Learning (RL) post-training across geographically distributed compute resources by fundamentally changing the synchronization unit. Instead of shipping massive full checkpoints (up to 500 GB), the proposed method uses a small 'lossless patch' representing only the changes in visible weights, shrinking the transfer size from hundreds of gigabytes to potentially hundreds of megabytes. This enables the rollout fleet to operate elastically across different regions and providers, decoupling it from the central training cluster.
Key takeaways
-
Decoupling RL Training from Compute Location
5:22
The standard RL loop couples the trainer and rollout worker in a single fast-fabric cluster. The solution proposes that the 'rollout serving island'—a coherent endpoint or local group of endpoints—is the movable unit, allowing it to operate across scattered, autoscaled capacity (the 'bazaar') rather than being restricted to one perfect cluster (the 'cathedral').
-
Sparse Weight Updates via Adam Absorption
8:05
The core mechanism relies on the fact that while gradients are dense, the actual change in the served weight view is extremely small. This 'Adam absorption' phenomenon occurs because a typical Adam step (around 3 millionths) is far smaller than the BF16 rounding boundary (around 0.0039), meaning the visible value does not change significantly.
-
Lossless Patch Synchronization
9:50
The synchronization unit is redefined as a 'lossless patch' (a diff) rather than a full checkpoint. This patch, which includes the change index and replacement bits, allows the rollout engine to bitwise reconstruct the exact served version from a much smaller object.
Technical details
-
BF16 View and Rounding Boundary
485s
The rollout engine serves a BF16 view. The rounding boundary is approximately $\theta/256$ (about 0.0039 for weights around 1). If the optimizer update step falls below this threshold, the visible weight value remains unchanged.
-
Adam Absorption Mechanism
485s
The Adam step is small and controlled. Because the typical post-training learning rate push is much smaller than the BF16 rounding boundary, the master weights can move without changing the value served by the rollout engine.
-
Stitch Implementation
790s
Modal's implementation, called Stitch, formalizes this process. On the trainer side, it publishes an immutable rollout weights version to a shared bulletin board. The rollout side then pulls these sparse deltas and performs weight sync across different regions and providers.
-
Low Precision Serving
630s
The concept extends to even lower precision formats like FP8, NF4, and MVF4. For instance, serving GLM 4.7 Air in FP8 showed only a 0.15% weight change on the first step, settling near 0.05%.
Mentioned resources
- Nan Jiang (Modal)
- Stitch
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.