# The AI Memory Wall: Why LLMs & AI Accelerators Hit a Bottleneck

## Executive summary

The video explains the 'Memory Wall,' a critical bottleneck in modern computing where the rapid growth of compute capabilities (e.g., B200 achieving 2,000 trillion operations per second) is outpacing the growth of memory bandwidth. Performance is limited not by raw compute power, but by the rate at which data can be fed to the chip. Understanding the workload's 'Arithmetic Intensity' and using the 'Roofline Model' are essential for diagnosing whether a system is compute-bound or memory-bound, guiding optimization efforts toward improving data transfer or computational efficiency.

## Key takeaways

- The Memory Wall Bottleneck: Compute capabilities have grown significantly (e.g., 6,500x growth from 2006 to 2026), but memory bandwidth has grown much slower, creating a widening gap that limits overall workload speed. This means a faster chip does not guarantee faster performance if data cannot be supplied quickly enough.
- Arithmetic Intensity is Key: This metric, defined as FLOPs per Byte (math operations per unit of data moved from memory), determines the bottleneck. High intensity suggests a compute-bound regime, while low intensity indicates a memory-bound regime.
- Batching Improves Efficiency: Batching, utilized by LLM serving systems like vLLM and SGLang, improves performance by allowing multiple requests to reuse the same weights. This effectively increases the arithmetic intensity, moving the workload closer to the chip's peak performance (the ridge point).

## Technical details

- The Memory Wall: The disparity between compute growth (measured in FLOPs per second) and memory bandwidth growth (measured in Bytes per second) is the Memory Wall. Additional compute power is useless if the data cannot be fed to the chip fast enough.
- Arithmetic Intensity & Roofline Model: The Roofline Model plots workload-dependent Arithmetic Intensity (FLOPs/Byte) against attainable bandwidth (FLOPs/second). The intersection of the chip's peak compute and memory bandwidth defines the 'ridge point,' which sets the physical limit for the workload.
- LLM Workload Analysis (Decode vs. Prefill): In LLMs, the 'decode' stage typically has low arithmetic intensity, placing it in the memory-bound regime. Conversely, the 'prefill' stage involves large GEM operations, resulting in higher arithmetic intensity and placing it in the compute-bound regime.

## Practical implications

- Before benchmarking, estimate the workload's Arithmetic Intensity to understand the fundamental bottleneck.
- Use the Roofline Model concept to compare the workload's arithmetic intensity against the hardware's peak memory bandwidth and peak compute.
- If memory-bound, optimization efforts must focus on improving data transfer speed (e.g., using techniques like batching).
- If compute-bound, optimization efforts should focus on improving computational efficiency or utilizing specialized accelerators.

## Topics

Memory Wall, Arithmetic Intensity, Roofline Model, LLM Inference, Compute-Bound, Memory-Bound, Batching, AI Accelerators, IBM Technology

Source: https://www.youtube.com/watch?v=4b6KpFN7Rio