# The Future of AI Infra: from Kubernetes to Agent Sandboxes — Akshat Bubna, Modal CTO

## Executive summary

Modal CTO Akshat Bubna discusses how modern AI workloads—especially those involving agents and elastic inference—exceed the design capabilities of traditional cloud infrastructure like Kubernetes. Modal addresses this by providing a specialized, highly elastic platform built on primitives such as serverless functions, GPU snapshotting, private IPv6 networking, and RDMA support. The discussion highlights a critical shift from optimizing for Developer Experience (DX) to enabling Agent Experience (AX), where the infrastructure must support complex, bursty compute patterns like those found in RL rollouts or custom model deployment across multiple cloud providers.

## Key takeaways

- Shift from DX to AX: Modal is shifting its focus from Developer Experience (DX) to Agent Experience (AX). This shift acknowledges that agents will increasingly operate on infrastructure, making the ability for an agent to self-provision and modify resources via decorators more valuable than manually writing complex YAML files in Kubernetes.
- K8s Limitations for AI: Traditional orchestration tools like Kubernetes were not designed for highly bursty, compute-heavy workloads common in AI. They struggle with the rapid scaling and specialized resource needs (like accelerators) required by modern models.
- Elastic Inference & Autoscaling: The platform excels at elastic inference for custom models (e.g., audio, video, robotics). The core challenge is managing unpredictable traffic patterns and scaling rapidly across different regions, a problem Modal addresses through its autoscaling primitives.
- Advanced Performance Primitives: To achieve frontier-level inference performance, Modal utilizes techniques like speculative decoding (using a smaller 'draft model' to predict tokens ahead of the main model) and DeFlash (a block-based speculator), achieving significant speedups without quality loss.

## Technical details

- Compute & Workloads: The platform supports diverse, bursty workloads including inference for custom models, multi-node training (for smaller scale post-training runs), and sandboxes. RL rollouts are cited as an example of extreme burstiness, potentially requiring 100,000 sandboxes.
- Networking & Connectivity: Modal provides advanced networking primitives including private IPv6 addressing (an overlay network for containers within a workspace) and RDMA support. The internal networking can achieve speeds up to 3 Tb per second, bypassing the standard TCP stack.
- Infrastructure Architecture: Modal operates as a 'supercloud' by building a capacity pool across 17 cloud providers. This strategy allows for high reliability and specialized resource allocation (e.g., data locality or low latency) without owning physical data centers.
- Agent & Code Execution: The system supports networked sandboxes and sidecars, allowing multiple containers within a single sandbox (a pod of containers). This provides granular control over outbound networking for complex agent tasks.
- Training Optimization: Auto Research is defined as an advanced hyperparameter sweep guided by model intuition, which is significantly more efficient than traditional search methods. The platform also supports automated internal processes like running Nvidia inside profiler to optimize configurations (e.g., switching between H200 and B200 GPUs).
- Security & Isolation: The architecture emphasizes hard guardrails for production agents, recognizing the risk of code execution and data exfiltration in complex AI workflows.

## Practical implications

- Companies building AI products must move beyond simple API calls and build complex backends that leverage specialized infrastructure primitives (like snapshotting and private networking) to achieve true product differentiation.
- The focus on Agent Experience suggests future cloud platforms will need robust, self-healing, and highly configurable runtime environments capable of handling autonomous code execution.
- For compute strategy, the ability to manage capacity across multiple clouds (17 providers) is a key differentiator for achieving superior reliability and cost efficiency.

## Topics

AI Infrastructure, Serverless Computing, Agent Orchestration, High-Performance Computing (HPC), Cloud Networking, Modal, DeFlash, speculative decoding, Private IPv6

Source: https://www.youtube.com/watch?v=UwxxlTNPjWo