Lecture 113: Every Microsecond Matters: Achieving Near Speed-of-Light Latency in GPU Collectives
This talk details advanced low-latency collective communication algorithms for GPU clusters, crucial for accelerating LLM inference. By replacing expensive global memory barriers with techniques like Sentinel synchronization and atomic operations, significant speedups are achieved.
Read article