# The Inference Frontier: from 100 to 10,000 tokens per second — Sean Lie, Cerebras CTO

## Executive summary

Cerebras CTO Sean Lie details the shift in AI infrastructure from merely increasing model size to achieving ultra-fast inference speeds. The core argument is that speed (throughput) itself enables entirely new classes of intelligent applications and agents. Cerebras showcased its CS4 wafer-scale architecture, demonstrating GPTOSS running at over 4,400 tokens per second (TPS). They previewed the next generation, CS5, which aims for up to 10,000 TPS on medium models and 5,000 TPS on frontier models. The industry is moving toward heterogeneous, disaggregated systems that integrate specialized components like OpenAI's Jalapeño chip to solve complex scaling challenges.

## Key takeaways

- Speed Defines Capability: What was once considered fast (100–200 TPS) is now viewed as 'batch mode.' Ultra-fast inference enables more reasoning loops and significantly more capable agents, transforming previously offline applications into real-time experiences.
- CS4 Performance Milestone: The CS4 architecture is a modular platform designed to bring wafer scale to hyperscale. It provides significantly more power and interconnect bandwidth, demonstrated by running GPTOSS at over 4,400 TPS.
- CS5 Roadmap (Preview): The next generation CS5 platform is designed for multiple generations of products. It aims to push performance further: up to 10,000 TPS for medium models (e.g., Gemma) and up to 5,000 TPS for frontier models (e.g., Gemini/DeepSeek).
- The Future is Heterogeneous & Disaggregated: Lie argues that the future of AI infrastructure requires integrating specialized components—such as prefill, attention, and KV cache loading—across different hardware types (e.g., wafer scale, SRAM-based chips like Jalapeño) to solve scaling challenges.

## Technical details

- Wafer Scale Architecture: Cerebras' modular platform is designed to bring wafer scale computing to hyperscale data centers. The architecture increases power delivery and interconnect bandwidth compared to previous generations.
- Inference Throughput Benchmarks: CS4 demonstrated GPTOSS running at over 4,400 TPS. CS5 is projected to reach up to 10,000 TPS (medium models) and 5,000 TPS (frontier models).
- AI-First Chip Design: OpenAI’s Jalapeño chip exemplifies an AI-first design methodology, which Lie argues is the future of the semiconductor industry. This approach enables significant gains in both throughput and latency.
- System Scaling Challenges: The necessity for wafer scale integration stems from the massive memory requirements of frontier models (e.g., trillions of parameters), which cannot be contained on single chips, necessitating advanced packaging and interconnect solutions.

## Practical implications

- The shift to ultra-fast inference enables the development of more complex and autonomous AI agents with multiple reasoning loops.
- Build engineers must focus on heterogeneous system integration (e.g., combining specialized compute, memory stacking, and high-bandwidth interconnects) rather than optimizing single components.
- The industry trend favors solutions that solve physical scaling limits through advanced packaging techniques like 3D DRAM stacking.

## Topics

AI Inference, Wafer Scale Computing, Semiconductor Architecture, High-Performance Computing (HPC), System Integration, CS4, CS5, Jalapeño chip

Source: https://www.youtube.com/watch?v=3uSI8q_RN-o