The Inference Frontier: from 100 to 10,000 tokens per second — Sean Lie, Cerebras CTO

Summary

Cerebras CTO Sean Lie details the shift in AI infrastructure from merely increasing model size to achieving ultra-fast inference speeds. The core argument is that speed (throughput) itself enables entirely new classes of intelligent applications and agents. Cerebras showcased its CS4 wafer-scale architecture, demonstrating GPTOSS running at over 4,400 tokens per second (TPS). They previewed the next generation, CS5, which aims for up to 10,000 TPS on medium models and 5,000 TPS on frontier models. The industry is moving toward heterogeneous, disaggregated systems that integrate specialized components like OpenAI's Jalapeño chip to solve complex scaling challenges.

Technical details

• Wafer Scale Architecture 195s

  Cerebras' modular platform is designed to bring wafer scale computing to hyperscale data centers. The architecture increases power delivery and interconnect bandwidth compared to previous generations.

• Inference Throughput Benchmarks 236s

  CS4 demonstrated GPTOSS running at over 4,400 TPS. CS5 is projected to reach up to 10,000 TPS (medium models) and 5,000 TPS (frontier models).

• AI-First Chip Design 760s

  OpenAI’s Jalapeño chip exemplifies an AI-first design methodology, which Lie argues is the future of the semiconductor industry. This approach enables significant gains in both throughput and latency.

• System Scaling Challenges 850s

  The necessity for wafer scale integration stems from the massive memory requirements of frontier models (e.g., trillions of parameters), which cannot be contained on single chips, necessitating advanced packaging and interconnect solutions.

Mentioned resources

• CS4 (Hardware Architecture)
• CS5 (Hardware Architecture (Preview))
• Jalapeño chip (AI Accelerator Chip)

Channel & topics

• Latent Space
• AI Inference
• Wafer Scale Computing
• Semiconductor Architecture
• High-Performance Computing (HPC)
• System Integration
• CS4
• CS5
• Jalapeño chip