# How AI Agents Work: Inside Agentic Inference

## Executive summary

This deep dive explains the shift from traditional, single-turn AI inference to complex, stateful agentic workflows. Agents perform work by reasoning, planning, and calling tools, leading to a compute workload that is significantly larger and more unpredictable than standard chat. To properly benchmark these systems, metrics must shift from simple tokens per second (TPS) to throughput per megawatt, interactivity, and end-to-end latency. Performance gains are achieved through system-level co-design, utilizing technologies like NVLink, NVIDIA Dynamo, and TensorRT-LLM to manage massive context growth and maximize resource utilization.

## Key takeaways

- Agentic Workflows vs. Traditional Chat: Traditional inference is a simple 'prompt in, answer out' cycle. Agentic workflows are complex, stateful, and unpredictable, involving model calls, tool calls, tool responses, and iterative reasoning. An agent's context can grow dramatically, consuming significantly more tokens (OpenRouter found one agentic request consumes roughly 15 times the tokens of ordinary chat) [00:00].
- Advanced Benchmarking Requirements: Traditional benchmarks (e.g., fixed 8,000 tokens in, 1,000 tokens out) are insufficient. Agentic benchmarks must replicate real-world production traces, preserving growing context, variable input/output lengths, and tool call gaps, ensuring the measured difference is due to the hardware/stack, not the test prompt [02:28].
- Holistic Performance Metrics: The optimal system is not simply the one with the highest raw throughput. Key metrics include: 1) Throughput per megawatt (efficiency), 2) Interactivity (tokens per second seen by the user), 3) End-to-end latency, and 4) Concurrency. The goal is maximizing throughput per megawatt while maintaining responsiveness [03:37].
- System Co-Design for Scale: Performance gains require co-design across the full system stack. Key components include: NVLink (connecting GPUs into a high-bandwidth scale-up domain), NVIDIA Dynamo (separating prefill from decode and reusing the KV cache), and TensorRT-LLM (adding NVF P4 precision and wide expert parallelism) [04:32].

## Technical details

- Agentic Workflow Mechanics: Agents operate by reasoning, planning, calling tools, running code, and interpreting results iteratively. A main agent can orchestrate subtasks by spinning up separate subagents (e.g., for searching a codebase), allowing the main agent's context to grow from tens of thousands to hundreds of thousands of tokens [00:48].
- Benchmark Methodology: Agentic benchmarks must replay real-world production traces, capturing variable input/output lengths and preserving tool call gaps, as the context can still occupy the Key-Value (KV) cache even when the agent is waiting [02:28].
- Hardware Performance Benchmarks: The GB300 NVL72 delivers up to 15 times higher throughput per megawatt than H200 NVL8 in agentic workloads. Furthermore, the Vera Rubin NVL72 can deliver up to 30 times higher AI factory throughput per megawatt than GB300 NVL72 on the Deep Seek V4 Pro workload, while maintaining target interactivity [04:32].

## Practical implications

- Build engineers must design inference infrastructure that accounts for massive, variable, and stateful context growth, moving beyond simple single-turn throughput metrics.
- System optimization requires integrating specialized hardware interconnects (NVLink) and distributed inference frameworks (NVIDIA Dynamo) to efficiently manage the KV cache and scale reasoning models.
- Performance evaluation must prioritize 'Throughput per Megawatt' alongside 'Interactivity' and 'End-to-End Latency' to ensure a positive user experience at scale.

## Topics

AI Agents, Agentic Inference, LLM Benchmarking, Inference Infrastructure, NVLink, NVIDIA Dynamo, TensorRT-LLM, Compute Workload, NVIDIA Vera Rubin and Blackwell Performance, Introducing NVIDIA Dynamo

Source: https://www.youtube.com/watch?v=d63Je4ZFBQA