The Future of AI Infra: from Kubernetes to Agent Sandboxes — Akshat Bubna, Modal CTO
Modal CTO Akshat Bubna discusses how modern AI workloads—especially those involving agents and elastic inference—exceed the design capabilities of traditional cloud infrastructure like Kubernetes. Modal addresses this by providing a specialized, highly elastic platform built on primitives such as serverless functions, GPU snapshotting, private IPv6 networking, and RDMA support. The discussion highlights a critical shift from optimizing for Developer Experience (DX) to enabling Agent Experience (AX), where the infrastructure must support complex, bursty compute patterns like those found in RL rollouts or custom model deployment across multiple cloud providers.
Key takeaways
-
Shift from DX to AX
17:33
Modal is shifting its focus from Developer Experience (DX) to Agent Experience (AX). This shift acknowledges that agents will increasingly operate on infrastructure, making the ability for an agent to self-provision and modify resources via decorators more valuable than manually writing complex YAML files in Kubernetes.
-
K8s Limitations for AI
4:19
Traditional orchestration tools like Kubernetes were not designed for highly bursty, compute-heavy workloads common in AI. They struggle with the rapid scaling and specialized resource needs (like accelerators) required by modern models.
-
Elastic Inference & Autoscaling
21:00
The platform excels at elastic inference for custom models (e.g., audio, video, robotics). The core challenge is managing unpredictable traffic patterns and scaling rapidly across different regions, a problem Modal addresses through its autoscaling primitives.
-
Advanced Performance Primitives
27:30
To achieve frontier-level inference performance, Modal utilizes techniques like speculative decoding (using a smaller 'draft model' to predict tokens ahead of the main model) and DeFlash (a block-based speculator), achieving significant speedups without quality loss.