# Why AI Compute Comes in Rectangles — Erik Bernhardsson, Modal

## Executive summary

Erik Bernhardsson of Modal discusses the unique challenges of building AI infrastructure, arguing that traditional cloud models (like three-year GPU capacity commitments) are ill-suited for the bursty, highly variable nature of modern AI workloads. Modal addresses this by providing a serverless compute platform that aggregates global GPU capacity, enabling rapid scaling for inference, training, and agent sandboxes. The discussion highlights the shift from traditional compute paradigms to highly granular, ephemeral, and composable building blocks.

## Key takeaways

- The Compute Paradigm Shift: AI workloads are not 'neat, three-year rectangles.' Modal's value proposition is aggregating global compute capacity to handle bursty, variable demand, which is a massive differentiator from traditional hyperscaler capacity models. (0:00, 5133)
- Sandboxes and Agent Workflows: Modal Sandboxes are evolving from simple 'lift and shift' dev boxes to highly ephemeral, single-command environments. The ability to rapidly spin up, snapshot, and resume sandboxes is key to enabling complex, parallelized agentic workflows and improving CI/CD efficiency. (41:21, 10:00)
- Infrastructure Primitives: The platform's strength lies in its primitives: Modal Functions (serverless compute for inference) and Sandboxes. The architecture is designed to manage capacity and isolation, allowing for high-density, multi-tenant usage across diverse workloads. (31:55, 10:00)
- Economic and Supply Constraints: The GPU market faces supply constraints, leading to long-term, high-cost capacity deals. While the market is highly speculative, Modal's model of continuous, on-demand capacity aggregation mitigates the risk associated with these large, fixed-term commitments. (13:21, 19:45)

## Technical details

- Compute Architecture: Modal offers a compute and storage platform enabling AI applications through a software SDK. Core services include Modal Functions (serverless compute for inference) and Modal Sandboxes (isolated execution environments). The platform manages capacity and scaling for inference, training/fine-tuning, and batch jobs. (0:00, 31:55)
- Scaling and Isolation: The system is built on primitives like container runtime and file system management. Key features include GPU snapshotting and a custom autoscaler, allowing for extremely fast scaling and efficient resource utilization, which is critical for bursty workloads. (10:00, 41:21)
- Agentic Workflows and CI/CD: Sandboxes are ideal for agentic workflows, enabling the parallelization of tasks (e.g., running every test in a CI suite in its own sandbox). The ability to minimize the overhead of starting and stopping sandboxes is crucial for realizing this massive parallelization potential. (41:21, 52:51)
- Security and Reliability: Security is addressed through isolated sandboxes. The company is actively using agents internally to probe for vulnerabilities and is exploring making security restrictions more restrictive by default, moving toward a 'zero trust' model. (46:48, 57:45)

## Practical implications

- For build engineers, the shift requires moving away from monolithic, steady-state infrastructure designs toward highly elastic, serverless, and ephemeral compute primitives.
- Focus on optimizing the 'transaction cost' (overhead) of starting and stopping isolated compute environments (sandboxes) to enable massive parallelization, especially in CI/CD and agentic systems.
- Architectural designs must account for the need to decouple services and prevent single points of failure, making the system resilient to rapid, unpredictable scaling demands.
- Consider adopting a multi-product, building-block approach (like Modal's vision) rather than focusing on a single, narrow service, as this increases general utility and market flexibility.

## Topics

AI Infrastructure, Serverless Compute, GPU Economics, Agentic Workflows, Cloud Architecture, CI/CD Optimization, Modal, Erik’s writing, Modal on serverless GPU economics

Source: https://www.youtube.com/watch?v=IiZ9yd2Do8I