How AI Agents Work: Inside Agentic Inference
Summary
This deep dive explains the shift from traditional, single-turn AI inference to complex, stateful agentic workflows. Agents perform work by reasoning, planning, and calling tools, leading to a compute workload that is significantly larger and more unpredictable than standard chat. To properly benchmark these systems, metrics must shift from simple tokens per second (TPS) to throughput per megawatt, interactivity, and end-to-end latency. Performance gains are achieved through system-level co-design, utilizing technologies like NVLink, NVIDIA Dynamo, and TensorRT-LLM to manage massive context growth and maximize resource utilization.
Key takeaways
-
Agentic Workflows vs. Traditional Chat
0:15
Traditional inference is a simple 'prompt in, answer out' cycle. Agentic workflows are complex, stateful, and unpredictable, involving model calls, tool calls, tool responses, and iterative reasoning. An agent's context can grow dramatically, consuming significantly more tokens (OpenRouter found one agentic request consumes roughly 15 times the tokens of ordinary chat) [00:00].
-
Advanced Benchmarking Requirements
3:48
Traditional benchmarks (e.g., fixed 8,000 tokens in, 1,000 tokens out) are insufficient. Agentic benchmarks must replicate real-world production traces, preserving growing context, variable input/output lengths, and tool call gaps, ensuring the measured difference is due to the hardware/stack, not the test prompt [02:28].
-
Holistic Performance Metrics
5:37
The optimal system is not simply the one with the highest raw throughput. Key metrics include: 1) Throughput per megawatt (efficiency), 2) Interactivity (tokens per second seen by the user), 3) End-to-end latency, and 4) Concurrency. The goal is maximizing throughput per megawatt while maintaining responsiveness [03:37].
-
System Co-Design for Scale
7:12
Performance gains require co-design across the full system stack. Key components include: NVLink (connecting GPUs into a high-bandwidth scale-up domain), NVIDIA Dynamo (separating prefill from decode and reusing the KV cache), and TensorRT-LLM (adding NVF P4 precision and wide expert parallelism) [04:32].
Technical details
-
Agentic Workflow Mechanics
48s
Agents operate by reasoning, planning, calling tools, running code, and interpreting results iteratively. A main agent can orchestrate subtasks by spinning up separate subagents (e.g., for searching a codebase), allowing the main agent's context to grow from tens of thousands to hundreds of thousands of tokens [00:48].
-
Benchmark Methodology
228s
Agentic benchmarks must replay real-world production traces, capturing variable input/output lengths and preserving tool call gaps, as the context can still occupy the Key-Value (KV) cache even when the agent is waiting [02:28].
-
Hardware Performance Benchmarks
432s
The GB300 NVL72 delivers up to 15 times higher throughput per megawatt than H200 NVL8 in agentic workloads. Furthermore, the Vera Rubin NVL72 can deliver up to 30 times higher AI factory throughput per megawatt than GB300 NVL72 on the Deep Seek V4 Pro workload, while maintaining target interactivity [04:32].
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.