# Lecture 113: Every Microsecond Matters: Achieving Near Speed-of-Light Latency in GPU Collectives

## Executive summary

This presentation details advanced techniques for achieving near speed-of-light latency in GPU collective communication operations, crucial for modern distributed workloads like LLM inference and scientific computing. The core challenge addressed is the high overhead of traditional global memory barriers during collectives (e.g., AllReduce). The proposed solutions—including Sentinel synchronization, Low Latency (LL) protocols, and the novel LL128 atomic algorithm—eliminate these expensive barriers by using data arrival itself as a synchronization signal. These advancements are packaged into an experimental device-side API to improve performance significantly in large-scale GPU clusters.

## Key takeaways

- Low Latency is Critical for Specific Regimes: Collective latency matters most when message sizes are small, the operation is repeated many times (e.g., auto-regressive decoding), and communication sits directly on the critical path.
- Global Memory Barriers are a Major Bottleneck: Traditional AllReduce implementations rely on global memory barriers, which can account for 40-50% of the total latency, even when optimizing the data movement itself.
- LL128 Atomic Algorithm Improves Scalability: The LL128 atomic algorithm leverages 128-byte cache line atomic additions over NVLink, offering superior scalability and reduced scratch buffer space compared to previous low-latency methods.
- API Abstraction Simplifies Implementation: A new experimental device-side API (Nickel LL Buffer) wraps these complex synchronization techniques (LL, Sentinel, etc.), allowing kernel writers to easily implement low-latency collectives without manually managing polling and buffer resets.

## Technical details

- Workload Analysis: Distributed LLM inference, particularly the auto-regressive decode phase, falls into the ideal latency regime: small message sizes (proportional to hidden size and batch size), high repetition count, and critical path communication.
- Communication Primitives: The work utilizes Symmetric Memory for direct remote device addressing within GPU kernels. Communication mechanisms include LSA (over PCIe/NVLink), NVLink switch features like multicast and in-network reduction, and GPU Initiated Networking (GEN).
- Low Latency Synchronization Techniques: To replace global barriers: 1) **LL Protocol:** Packs data with a flag for atomic writing. Simple, but wastes 50% bandwidth. 2) **Sentinel Sync:** Uses special sentinel values to poll for data arrival (100% bandwidth utilization). Requires careful buffer initialization/resetting. 3) **Double Buffering + Implicit Sync:** Achieves synchronization by using bidirectional communication flow control signals.
- LL128 Atomic Algorithm: This advanced technique uses 128-byte cache line atomic additions over NVLink. It allows ranks to atomically add contributions directly to a shared destination, eliminating the need for local reduction stages and significantly reducing required scratch buffer space.

## Practical implications

- By optimizing collective communication from microseconds to near speed-of-light latency, end-to-end inference throughput can see significant improvements.
- The performance gains translate directly into cost savings for cloud providers running LLM inference workloads (estimated at $2-$2.3 per million output tokens in the tested setup).
- The API provides a reusable building block for custom kernel fusion, allowing system builders to integrate low-latency communication primitives easily.

## Topics

GPU Computing, Distributed Systems, Machine Learning (LLMs), High Performance Computing (HPC), Communication Protocols, https://arxiv.org/abs/2607.16100

Source: https://www.youtube.com/watch?v=J7-uvBSG7ho