Serving World Models: Latency vs Throughput with NVIDIA Cosmos 3 Super on Nebius | Cosmos Labs
Summary
This technical deep dive examines the complex trade-offs between latency and throughput when serving large-scale world foundation models, specifically NVIDIA Cosmos 3 Super, for video generation. The discussion, featuring Nebius Physical AI team members, benchmarks the model's performance using vLLM-Omni across various GPU topologies (e.g., one eight-GPU replica vs. eight single-GPU replicas). Key findings emphasize that the optimal serving configuration depends entirely on the use case: low latency is critical for interactive/real-time applications, while high throughput is better for batch data generation. The presentation also details advanced optimization techniques like tensor, sequence, and hybrid data parallelism.
Key takeaways
-
Latency vs. Throughput Trade-off
20:50
For video generation, the choice between a single, large replica (high coordination) and multiple, independent replicas (high concurrency) is critical. While a single 8-GPU replica can achieve very low latency (e.g., 72 seconds for a 7-second clip), running eight individual replicas can result in a significantly higher throughput (more video generated per node-hour) despite a higher latency (e.g., 382 seconds for the same clip).
-
Model Scaling and Architecture
6:40
The Cosmos 3 Super is a 64 billion parameter world foundation model designed for high fidelity and physics accuracy. The family also includes Cosmos 3 Nano (16B) and Cosmos 3 Edge (4B) for different deployment needs.
-
Benchmarking Best Practices
30:00
To ensure measurable results, experiments must establish a control set by fixing variables such as the software (vLLM-Omni), model version (Cosmos 3 pin version), video quality (720p), and prompt/negative prompt. Guardrails should be explicitly managed, as enabling them can skew performance results.
-
Advanced Parallelism Techniques
35:00
Serving large models requires understanding various parallelism methods: Tensor Parallelism (sharing diffusion transformer weights across GPUs), Classifier Free Guidance (CFG) parallelism, Sequence Parallelism (splitting the video token sequence), and Hybrid Shared Data Parallelism (sharding model weights across GPUs).
Technical details
-
World Foundation Models (World Models)
350s
These models predict physical world interactions (e.g., how robots operate) by understanding how the world moves and interacts, moving beyond traditional LLM text prediction.
-
Cosmos 3 Super Architecture
450s
Cosmos 3 utilizes a multimodal omnimodal approach, ingesting images, videos, actions, and text. The process involves an autoregressive transformer (for reasoning) followed by a diffusion transformer (the generator) to produce video/sound/action.
-
Optimization and Quantization
2250s
The discussion covered optimizing for GPU performance and memory efficiency, including techniques like using lower precision formats (e.g., FP8) and quantization, though the primary benchmarks were run using BF16.
-
Deployment Infrastructure
2600s
Nebius provides the FISCO AI workbench, an open-source framework acting as an agent control plane to simplify deployment infrastructure setup (e.g., Kubernetes configuration) for running world model benchmarks.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.