# Serving World Models: Latency vs Throughput with NVIDIA Cosmos 3 Super on Nebius | Cosmos Labs

## Executive summary

This technical deep dive examines the complex trade-offs between latency and throughput when serving large-scale world foundation models, specifically NVIDIA Cosmos 3 Super, for video generation. The discussion, featuring Nebius Physical AI team members, benchmarks the model's performance using vLLM-Omni across various GPU topologies (e.g., one eight-GPU replica vs. eight single-GPU replicas). Key findings emphasize that the optimal serving configuration depends entirely on the use case: low latency is critical for interactive/real-time applications, while high throughput is better for batch data generation. The presentation also details advanced optimization techniques like tensor, sequence, and hybrid data parallelism.

## Key takeaways

- Latency vs. Throughput Trade-off: For video generation, the choice between a single, large replica (high coordination) and multiple, independent replicas (high concurrency) is critical. While a single 8-GPU replica can achieve very low latency (e.g., 72 seconds for a 7-second clip), running eight individual replicas can result in a significantly higher throughput (more video generated per node-hour) despite a higher latency (e.g., 382 seconds for the same clip).
- Model Scaling and Architecture: The Cosmos 3 Super is a 64 billion parameter world foundation model designed for high fidelity and physics accuracy. The family also includes Cosmos 3 Nano (16B) and Cosmos 3 Edge (4B) for different deployment needs.
- Benchmarking Best Practices: To ensure measurable results, experiments must establish a control set by fixing variables such as the software (vLLM-Omni), model version (Cosmos 3 pin version), video quality (720p), and prompt/negative prompt. Guardrails should be explicitly managed, as enabling them can skew performance results.
- Advanced Parallelism Techniques: Serving large models requires understanding various parallelism methods: Tensor Parallelism (sharing diffusion transformer weights across GPUs), Classifier Free Guidance (CFG) parallelism, Sequence Parallelism (splitting the video token sequence), and Hybrid Shared Data Parallelism (sharding model weights across GPUs).

## Technical details

- World Foundation Models (World Models): These models predict physical world interactions (e.g., how robots operate) by understanding how the world moves and interacts, moving beyond traditional LLM text prediction.
- Cosmos 3 Super Architecture: Cosmos 3 utilizes a multimodal omnimodal approach, ingesting images, videos, actions, and text. The process involves an autoregressive transformer (for reasoning) followed by a diffusion transformer (the generator) to produce video/sound/action.
- Optimization and Quantization: The discussion covered optimizing for GPU performance and memory efficiency, including techniques like using lower precision formats (e.g., FP8) and quantization, though the primary benchmarks were run using BF16.
- Deployment Infrastructure: Nebius provides the FISCO AI workbench, an open-source framework acting as an agent control plane to simplify deployment infrastructure setup (e.g., Kubernetes configuration) for running world model benchmarks.

## Practical implications

- When designing a physical AI system, the build engineer must first define the primary operational goal: Is the system latency-sensitive (requiring low-latency, potentially single-replica setups) or is it batch-processing intensive (favoring high-throughput, multi-replica setups)?
- Use the provided benchmarking framework (nebius-physical-ai) to establish a controlled baseline, ensuring all variables (prompt, seed, resolution, guardrails) are fixed for repeatable results.
- For resource allocation, consider that while high-end nodes (HGX B200/H200) are ideal, alternative cloud compute options like serverless or preemptible VMs can be used for initial, cost-effective experimentation.

## Topics

World Models, Video Generation, GPU Optimization, Distributed Computing, Physical AI, Benchmarking, Technical blog (Serving World Models), Reproduce the benchmarks, NVIDIA Cosmos platform

Source: https://www.youtube.com/watch?v=i9FZyG9-Gx0