# Building Reliable AI Infrastructure at Scale with Extreme Co-design

## Executive summary

The presentation details how building reliable, next-generation AI infrastructure requires 'extreme co-design' across compute, networking, storage, and software. NVIDIA's Vera Rubin platform is presented as an integrated AI factory, moving beyond single components to provide a scalable, resilient, and highly efficient ecosystem. Key advancements include the shift from traditional Generative AI to complex Agentic AI, which demands specialized CPUs (Vera CPU) and sophisticated, multi-plane networking (Spectrum-X Ethernet) to ensure high throughput and fault tolerance at massive scale.

## Key takeaways

- The Shift to Agentic AI: AI workloads are evolving from simple question-answering (Generative AI) to complex Agentic AI, which involves multi-step reasoning, tool-calling, and continuous context management. This requires a platform that supports complex, multi-tasking workflows, not just single-pass inference.
- Co-Design for Reliability and Scale: The Vera Rubin platform is an integrated AI factory designed for co-design across seven chips and five racks. This holistic approach ensures that components (compute, networking, storage) work together seamlessly, maximizing efficiency and minimizing downtime.
- Advanced Networking and Resilience: The architecture utilizes Multi-plane topology and technologies like Spectrum-X Ethernet with co-packaged optics (CPO) to achieve massive scale (supporting 512,000 GPUs) while maintaining low latency and high resilience. Hardware-level fault detection and recovery can occur in milliseconds, preventing application timeouts.
- Efficiency through Quantization and Sparsity: The platform supports advanced techniques like NVFP4 (4-bit quantization) combined with sparsity features, enabling models to achieve quality comparable to 16-bit models while significantly boosting performance and efficiency.

## Technical details

- AI Infrastructure Architecture: NVIDIA positions itself not just as a chip manufacturer, but as an AI infrastructure builder, defining a five-layer stack: Power, Chips/Hardware, Infrastructure, Models, and Applications. The entire stack is built upon the CUDA foundation.
- Vera Rubin Platform Components: The platform consists of seven chips across five racks, including the Vera Rubin NVL 72 (for training/inference), Groq LPX (for inference acceleration), Vera CPU (for tool-calling/RL workloads), Vera BlueField 4 STX (storage), and Spectrum-X Ethernet (networking).
- Performance Metrics (100 MW Factory): A 100 MW Vera Rubin factory can achieve inference capacity of up to 2 Zettaflops (NVFB4) with fast memory up to 42 Petabytes, providing flexibility for training, inference, and post-training workloads.
- CPU Performance for Agentic AI: The Vera CPU is optimized for Agentic AI, offering double the single-thread performance of current accelerators. Key features include 88 cores on a single die and a second-generation fabric providing 3.4 Tb/s bandwidth, crucial for coordinating multiple agents and maintaining context.
- Software and Deployment (DSX): DSX is the software platform for designing and deploying the entire AI factory. DSX Max LPS can reallocate unused power across racks, increasing GPU capacity by up to 40% without increasing power budget.

## Practical implications

- Build engineers can leverage the DSX platform to significantly reduce design time and integration risks by using reference blueprints for compute, networking, and storage.
- The focus on hardware-level fault detection and recovery (e.g., in networking) minimizes downtime and prevents costly application rollbacks, improving overall system reliability.
- The modularity of the Vera Rubin design (e.g., easily replaceable/reconfigurable compute sleds) allows for rapid deployment and adaptation to evolving AI workloads.
- Adopting 4-bit quantization (NVFP4) with sparsity allows for higher performance density and better utilization of expensive GPU resources.

## Topics

AI Infrastructure, GPU Computing, Networking, System Reliability, Agentic AI, Co-design, NVIDIA Vera Rubin, CUDA, NVLink, Spectrum-X Ethernet, A100 / H100, DSX (NVIDIA Software)

Source: https://www.youtube.com/watch?v=FIOXn-oPvpA