# The Frontier AI Inference Cloud for Agents — Byung-Gon (Gon) Chun, FriendliAI

## Executive summary

The talk outlines the shift from traditional request-based LLM inference (chat) to complex, task-based agentic inference. Because agents run in long loops (plan, act, observe, repeat) and generate massive context prefixes, standard inference stacks fail. FriendliAI addresses this by presenting an 'inference cloud for agents' built on four pillars: Prefix Caching, hierarchical KV Cache Management, Cache Aware Routing, and Agent Aware Scheduling. This architecture enables open-weight models (like GLM 5.2) to achieve frontier quality at a fraction of the cost of closed models, making complex, multi-step agents economically viable for production.

## Key takeaways

- The Unit of Work Shifts from Request to Task: In chat, the unit is the request (optimizing for low latency). In agents, the unit is the task, which runs autonomously for minutes or hours. Optimization must therefore focus on end-to-end task latency, not single-call latency. (3:18)
- Open Weights Achieve Frontier Quality at Lower Cost: Open-weight models are now capable of completing complex agentic tasks (e.g., building a tower defense game) at a usable level, rivaling closed models. For the same task, GLM 5.2 (open-weight) was approximately 5.6 times cheaper than Anthropic's Opus 4.8. (2:06)
- The Four Pillars of Agentic Inference Cloud: To optimize for task latency, the stack relies on: 1) Prefix Caching (computing shared prefixes once); 2) Hierarchical KV Cache Management (across GPU, host, and disk); 3) Cache Aware Routing (sending requests to replicas holding the prefix); and 4) Agent Aware Scheduling (understanding the long-term program context). (7:02)

## Technical details

- Prefix Caching: Since consecutive agent steps share enormous prefixes, the system computes the Key-Value (KV) cache for this prefix once and reuses it, significantly reducing compute spent on already-done work. (7:30)
- KV Cache Management: The system uses frugal memory management and KV cache contigation to pack context onto GPU memory. It implements hierarchical caching across GPU memory, host memory, and disk to scale beyond GPU limits. (8:15)
- Cache Aware Routing: Unlike naive load balancers, a cache aware router sends a request to a specific replica that already holds the required prefix cache, turning a costly prefill into a single cache hit while maintaining load balance. (9:19)
- Agentic Workload Structure: Agent workflows are iterative loops: Plan (LLM call) $\rightarrow$ Act (Tool call) $\rightarrow$ Observe (Context update) $\rightarrow$ Repeat. The context grows continuously, and the number of model calls is variable, making fixed request rate planning impossible. (4:00)

## Practical implications

- Build engineers can integrate open-weight frontier models (e.g., GLM 5.2) into existing agent stacks, achieving high capability with significantly reduced operational costs.
- The FriendliAI platform offers three deployment options: Model API (fastest start), Dedicated Endpoints (guaranteed SLAs), and BYOG (Bring Your Own GPU) for maximum control.
- The focus on end-to-end task latency allows for the deployment of more reliable and complex production agents compared to systems optimized only for single-request speed.

## Topics

AI Agents, LLM Inference, Cloud Computing, Open Weights, Distributed Systems, FriendliAI, GLM 5.2, Anthropic Opus 4.8

Source: https://www.youtube.com/watch?v=Hvb2LfMH58c