The Future of AI Infra: from Kubernetes to Agent Sandboxes — Akshat Bubna, Modal CTO
Summary
Modal CTO Akshat Bubna discusses how modern AI workloads—especially those involving agents and elastic inference—exceed the design capabilities of traditional cloud infrastructure like Kubernetes. Modal addresses this by providing a specialized, highly elastic platform built on primitives such as serverless functions, GPU snapshotting, private IPv6 networking, and RDMA support. The discussion highlights a critical shift from optimizing for Developer Experience (DX) to enabling Agent Experience (AX), where the infrastructure must support complex, bursty compute patterns like those found in RL rollouts or custom model deployment across multiple cloud providers.
Key takeaways
-
Shift from DX to AX
17:33
Modal is shifting its focus from Developer Experience (DX) to Agent Experience (AX). This shift acknowledges that agents will increasingly operate on infrastructure, making the ability for an agent to self-provision and modify resources via decorators more valuable than manually writing complex YAML files in Kubernetes.
-
K8s Limitations for AI
4:19
Traditional orchestration tools like Kubernetes were not designed for highly bursty, compute-heavy workloads common in AI. They struggle with the rapid scaling and specialized resource needs (like accelerators) required by modern models.
-
Elastic Inference & Autoscaling
21:00
The platform excels at elastic inference for custom models (e.g., audio, video, robotics). The core challenge is managing unpredictable traffic patterns and scaling rapidly across different regions, a problem Modal addresses through its autoscaling primitives.
-
Advanced Performance Primitives
27:30
To achieve frontier-level inference performance, Modal utilizes techniques like speculative decoding (using a smaller 'draft model' to predict tokens ahead of the main model) and DeFlash (a block-based speculator), achieving significant speedups without quality loss.
Technical details
-
Compute & Workloads
2365s
The platform supports diverse, bursty workloads including inference for custom models, multi-node training (for smaller scale post-training runs), and sandboxes. RL rollouts are cited as an example of extreme burstiness, potentially requiring 100,000 sandboxes.
-
Networking & Connectivity
2045s
Modal provides advanced networking primitives including private IPv6 addressing (an overlay network for containers within a workspace) and RDMA support. The internal networking can achieve speeds up to 3 Tb per second, bypassing the standard TCP stack.
-
Infrastructure Architecture
2305s
Modal operates as a 'supercloud' by building a capacity pool across 17 cloud providers. This strategy allows for high reliability and specialized resource allocation (e.g., data locality or low latency) without owning physical data centers.
-
Agent & Code Execution
2375s
The system supports networked sandboxes and sidecars, allowing multiple containers within a single sandbox (a pod of containers). This provides granular control over outbound networking for complex agent tasks.
-
Training Optimization
2685s
Auto Research is defined as an advanced hyperparameter sweep guided by model intuition, which is significantly more efficient than traditional search methods. The platform also supports automated internal processes like running Nvidia inside profiler to optimize configurations (e.g., switching between H200 and B200 GPUs).
-
Security & Isolation
1340s
The architecture emphasizes hard guardrails for production agents, recognizing the risk of code execution and data exfiltration in complex AI workflows.
Mentioned resources
- Modal
- DeFlash
- speculative decoding
- Private IPv6
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.