Ask the Experts: Nemotron 3.5 Super VL | Nemotron Labs

Summary

NVIDIA introduced Nemotron 3.5 Super VL, a multimodal open model designed for high-volume, specialized autonomous agent tasks. The model processes both text and images and is optimized for efficiency, boasting 120 billion total parameters while maintaining performance comparable to much larger models. Key architectural advancements include the integration of vision capabilities and sophisticated training techniques like distillation and mid-training, allowing for superior performance in agentic and vision-language workloads. The model is optimized for deployment on hardware like DGX Spark and utilizes NVFP4 quantization for efficient local operation.

Technical details

• Multimodal Architecture (VL) 420s

  The vision capability was added later in the development cycle. The process involved adding a vision model and a vision projection layer alongside the LLM, followed by specialized vision pre-training to integrate the modalities.

• Training Methodology (3 to 3.5) 1300s

  The upgrade from Super 3 to 3.5 involved multiple stages: 1) Distillation from the Ultra model during pre-training, 2) A newly added mid-training stage (the first for Nemotron models), and 3) Scaling up the quality and quantity of Supervised Fine-Tuning (SFT) traces. High-quality data filtering was critical to maintain performance despite a smaller token count compared to Ultra.

• Agentic and RL Training 1400s

  The model benefits from extensive Reinforcement Learning (RL) with sandboxes and execution. For agentic tasks, the team scaled up the RL process, focusing on 'long horizon RL' for complex tasks like search and coding, which is crucial for robust agent behavior.

• Mixture of Experts (MoE) Balance 1800s

  The experts are not domain-specific but are token-granular. Maintaining 'expert load balancing' during training (SFT, pre-training, mid-training) is critical. Unbalanced experts can lead to instability and 'train-inference mismatch' during RL, making balanced training essential for robust performance.

• Inference Modes and Latency 2400s

  The model offers three reasoning modes (Off, Medium, High). For optimal usability, the recommendation is to start with 'Reasoning Off' for day-to-day tasks, escalating to Medium, and finally High if accuracy is insufficient. Training aims for calibration, ensuring the model reasons appropriately without excessive token usage.

Mentioned resources

• GitHub (nvidia-nemotron) (Code/Recipes)
• Technical Report (Documentation)
• Learning Paths (Educational Resource)

Channel & topics

• NVIDIA Developer
• Multimodal AI
• Large Language Models (LLMs)
• Vision-Language Models (VLMs)
• Agentic Workflows
• Model Quantization (NVFP4)
• Reinforcement Learning (RL)
• GitHub (nvidia-nemotron)
• Technical Report
• Learning Paths