# Lecture 113: Every Microsecond Matters: Achieving Near Speed-of-Light Latency in GPU Collectives

## Executive summary

The lecture details advanced low-latency collective communication algorithms for GPU clusters, crucial for accelerating large language model (LLM) inference. The core challenge addressed is that traditional global memory synchronization barriers are significant bottlenecks in auto-regressive decoding phases. Solutions involve novel techniques—such as Low Latency (LL) protocols, Sentinel synchronization, and Double Buffering—that replace explicit barriers by using data arrival itself as the synchronization signal. Furthermore, a new `LL128 atomic` algorithm is introduced to achieve highly scalable, low-overhead reduction operations.

## Key takeaways

- Latency Bottleneck in LLM Inference: During auto-regressive decoding (the decode phase), message sizes are small, and the collective operation is executed repeatedly on the critical path. Saving microseconds per AllReduce can lead to substantial end-to-end speedups [0:53].
- Symmetric Memory for Remote Access: Symmetric memory allows a GPU kernel to directly calculate and address the corresponding location of an object on another participating GPU, simplifying remote device memory access within kernels [1:40].
- Eliminating Global Barriers: The primary bottleneck in traditional AllReduce algorithms (like OneShot or TwoShot) is the expensive global memory barrier. New techniques replace these barriers by using data arrival as an implicit synchronization signal, trading space for latency [2:36].
- The LL128 Atomic Algorithm: A novel approach utilizes atomic additions over MVLink to perform reductions directly into a shared destination buffer. This significantly reduces required scratch buffer space and improves scalability for larger numbers of ranks compared to previous methods [4:30].

## Technical details

- Collective Communication Phases: Transformer layers involve AllReduce operations. The prefill phase is bandwidth-sensitive (proportional to total sequence length and hidden size), while the decode phase is latency-critical, as message sizes are small and the operation runs repeatedly [0:15].
- Low Latency Synchronization Techniques: Three main techniques eliminate global barriers: 1) **LL Protocol:** Packs data with a flag for atomic writing. 2) **Sentinel Synchronization:** Initializes buffers with a unique sentinel value (e.g., -N), allowing data arrival to signal completion. 3) **Double Buffering/Implicit Sync:** Uses bidirectional communication flow control, where receiving data acts as the 'credit' or synchronization signal for the next iteration [3:50].
- Hardware and APIs: The work leverages hardware features like Symmetric Memory, LSA (Low Store Accessible Memory) over PCIe/MVLink, and advanced interconnects like MVSwitch for multicast operations. The proposed solution is an experimental device-side API built on existing NVIDIA NCCL APIs to abstract these low-latency mechanisms [1:20].
- Performance Benchmarks: Microbenchmarks show that the new low-latency kernels (e.g., OneShot with Sentinel/LL) significantly outperform existing implementations across small and medium message sizes, especially when utilizing multicast capabilities [4:50]. The performance gain translates to measurable cost savings in real-world LLM inference workloads (e.g., $2-$3 per million output tokens for DeepCarOne) [6:15].

## Practical implications

- Enables faster, more efficient LLM inference by optimizing the critical path of auto-regressive decoding.
- The proposed API abstracts complex low-latency synchronization logic (LL, Sentinel) into reusable building blocks for custom kernel development.
- Demonstrates that microsecond improvements in collective operations can translate to significant cost savings when running large-scale cloud inference services.

## Topics

GPU Computing, Machine Learning Infrastructure, High Performance Computing (HPC), Collective Communication, Low Latency Networking, arXiv:2607.16100

Source: https://www.youtube.com/watch?v=TZnJYRTSGVk