# Taking Reinforcement Learning Cross Datacenter — Nan Jiang, Modal

## Executive summary

This talk addresses scaling Reinforcement Learning (RL) post-training across geographically distributed compute resources by fundamentally changing the synchronization unit. Instead of shipping massive full checkpoints (up to 500 GB), the proposed method uses a small 'lossless patch' representing only the changes in visible weights, shrinking the transfer size from hundreds of gigabytes to potentially hundreds of megabytes. This enables the rollout fleet to operate elastically across different regions and providers, decoupling it from the central training cluster.

## Key takeaways

- Decoupling RL Training from Compute Location: The standard RL loop couples the trainer and rollout worker in a single fast-fabric cluster. The solution proposes that the 'rollout serving island'—a coherent endpoint or local group of endpoints—is the movable unit, allowing it to operate across scattered, autoscaled capacity (the 'bazaar') rather than being restricted to one perfect cluster (the 'cathedral').
- Sparse Weight Updates via Adam Absorption: The core mechanism relies on the fact that while gradients are dense, the actual change in the served weight view is extremely small. This 'Adam absorption' phenomenon occurs because a typical Adam step (around 3 millionths) is far smaller than the BF16 rounding boundary (around 0.0039), meaning the visible value does not change significantly.
- Lossless Patch Synchronization: The synchronization unit is redefined as a 'lossless patch' (a diff) rather than a full checkpoint. This patch, which includes the change index and replacement bits, allows the rollout engine to bitwise reconstruct the exact served version from a much smaller object.

## Technical details

- BF16 View and Rounding Boundary: The rollout engine serves a BF16 view. The rounding boundary is approximately $\theta/256$ (about 0.0039 for weights around 1). If the optimizer update step falls below this threshold, the visible weight value remains unchanged.
- Adam Absorption Mechanism: The Adam step is small and controlled. Because the typical post-training learning rate push is much smaller than the BF16 rounding boundary, the master weights can move without changing the value served by the rollout engine.
- Stitch Implementation: Modal's implementation, called Stitch, formalizes this process. On the trainer side, it publishes an immutable rollout weights version to a shared bulletin board. The rollout side then pulls these sparse deltas and performs weight sync across different regions and providers.
- Low Precision Serving: The concept extends to even lower precision formats like FP8, NF4, and MVF4. For instance, serving GLM 4.7 Air in FP8 showed only a 0.15% weight change on the first step, settling near 0.05%.

## Practical implications

- Enables RL training to use globally scattered GPU capacity (elastic rollout fleet), removing the dependency on a single, tightly coupled cluster.
- Significantly reduces network bandwidth requirements for weight updates by replacing full checkpoint transfers (500 GB) with sparse delta patches (hundreds of MB).
- The architecture allows the trainer to remain in its high-speed cluster while the rollout fleet operates independently across diverse regions and providers.

## Topics

Reinforcement Learning, Distributed Systems, Model Quantization, Weight Synchronization, Large Language Models (LLMs), Nan Jiang (Modal), Stitch

Source: https://www.youtube.com/watch?v=maRzp4kImJ4