NVIDIA Dynamo — The Video Signal

Aug 27, 2026 · 49:23

Ask the Experts: How NeMo Switchyard Helps Agents Select Models | Nemotron Labs

NeMo Switchyard is an open-source model routing library designed for AI agents to solve the problem of relying on a single monolithic LLM. It automatically routes each agent query or step to the optimal model—selecting from any combination of local/cloud and open/closed models—based on real-time needs, optimizing for accuracy, cost, and latency. The system operates beyond simple request routing by tracking state across multi-turn agentic workflows, making it a critical component for building robust, efficient AI systems.

Key takeaways

System of Models Approach 2:49
The industry is moving away from the 'one model to rule them all' concept toward a 'system of models,' where multiple specialized models are used for different tasks, improving efficiency and capability (0:02:45).
Agent-Aware Routing vs. Simple Routing 5:18
Switchyard is more than a simple router; it operates on an agentic workflow, tracking state (e.g., tool calls, message history) across multi-turn sessions to make intelligent model selection decisions (0:04:25).
Optimization and Learning 23:18
The system treats model selection as an optimization problem. It can learn by analyzing agent traces and behavior, predicting potential errors or resource needs to route proactively and save tokens/time (0:21:58).
Full-Stack Routing Flywheel 6:10
The roadmap envisions a full 'flywheel' of routing, connecting model selection to inference optimization (via NVIDIA Dynamo) and data privacy/anonymization. This allows for continuous improvement across the entire agent lifecycle (0:06:10).

Watch on YouTube Full article

Aug 3, 2026 · 1:42:54

The Inference Frontier: 10x Faster Models to Self-Optimizing AI — Philip Kiely & Ali Taha, Baseten

The video provides a deep dive into modern inference engineering, detailing how large language models (LLMs) are optimized for production use to achieve massive speedups (up to 10x). Key techniques discussed include cache-aware routing, disaggregated prefill/decode, speculative decoding, and advanced quantization methods like NVFP4. The conversation also expands beyond LLMs into video generation architectures and the convergence of training and inference, emphasizing that modern AI infrastructure is becoming a complex blend of software optimization (kernels, runtimes) and hardware architecture (B200s, Blackwell).

Key takeaways

Inference Optimization Stacking for 10x Gains 1:39:42
Achieving massive speedups requires stacking multiple optimizations. A typical baseline might be 30-40 tokens/second; reaching 10x performance involves combining techniques like speculative decoding, quantization (e.g., BF-16 to NVFP4), cache-aware routing, and disaggregation of prefill/decode stages. The overall gain is multiplicative.
The Role of Specialized Tooling (Dynamo) 1:22:40
NVIDIA Dynamo is presented not as an out-of-the-box optimizer, but as a developer toolkit/library for coordinating complex inference tasks like KV cache offloading and prefill/decode disaggregation across heterogeneous hardware.
Training vs. Inference Convergence 1:40:52
The field is moving toward a continuous improvement loop where inference data (traces) are used to continually post-train, fine-tune, and optimize models. This unification means inference engineers must be highly familiar with training techniques like Quantization Aware Training (QAT).
Video Generation Complexity 1:30:02
Unlike text generation (which is autoregressive), video diffusion models are non-sequential. Generating long, coherent videos requires managing massive attention spans (e.g., attending to tens of thousands of tokens for just 5 seconds), making the quadratic complexity a major bottleneck.

Watch on YouTube Full article