The Video Signal

Topic

Model Optimization

All digests tagged Model Optimization

Ask the Experts: Inside Nemotron Post-Training | Nemotron Labs thumbnail

· 51:07

Ask the Experts: Inside Nemotron Post-Training | Nemotron Labs

This session details the advanced post-training pipeline used by NVIDIA AI researchers to build state-of-the-art (SOTA) models like Nemotron, focusing on enhancing intelligence and enabling agentic capabilities. Post-training is presented as an evolution from traditional task-specific training, utilizing structured data, chat templates, and specialized frameworks like NeMo Gym and NeMo RL. Key strategies discussed include performing ablation studies on data subsets, employing Mixture of Experts (MoE) for capability generalization, and establishing robust feedback loops using real-world user data to prevent model degradation.

Key takeaways

  1. Post-Training Evolution 3:58

    Post-training builds upon pre-training (which uses massive, diverse, unstructured data for causal language modeling) by focusing on structured data. This teaches the model to follow instructions, use chat templates, and emit tool calls, moving beyond simple task-specific or multi-task training.

  2. Structured Data for Capabilities 7:29

    To narrow model focus, post-training emphasizes structured data (e.g., tool responses, user prompts, tool sets) over unstructured text. This allows the model to learn specific formats, such as JSON or XML, for tool interaction.

  3. Mitigating Capability Degradation 13:54

    To specialize a model (e.g., for software engineering) without losing general capabilities, techniques include including general chat data in the blend and using algorithmic approaches like multi-teacher on-policy distillation (MOPD).

  4. Data Bias and Profiling 19:20

    Data quality is paramount. Researchers must analyze data statistics (e.g., trajectory length, tool call diversity) and perform ablation studies to identify and address biases (e.g., over-reliance on a single tool) that could cause model failure in real-world use.

  5. Starting the Pipeline 25:12

    For developers, the process involves defining the model's 'soul' (core capabilities), quantifying these goals via benchmarks, setting quantitative milestones, and iterating through the post-training pipeline. Starting resources include Nemotron 3.5 Lightning.

Watch on YouTube Full article

From Video to Voice: Build Faster with TensorRT Model Connect thumbnail

· 32:38

From Video to Voice: Build Faster with TensorRT Model Connect

TensorRT Model Connect (TRT MC) is a feature designed to drastically simplify the deployment of open-source AI models into production applications. It provides a consistent, low-overhead workflow that allows developers to convert models (e.g., PyTorch checkpoints) into optimized, deployable 'bundles.' This process handles model analysis, optimization (including graph fusion and tactic selections), and runtime generation, enabling fast, end-to-end inference across diverse model types and hardware configurations, including multi-device scaling.

Key takeaways

  1. Simplified Model Deployment Pipeline 2:00

    TRT MC streamlines the process from open-source model to deployable application. Developers use a simple Python command to convert a model checkpoint into a deployable bundle, abstracting away complex pipeline plumbing (e.g., auto-regressive loops).

  2. Support for Diverse AI Workloads 5:40

    The tool supports a wide range of model architectures beyond LLMs, including audio generation (Bark, Nvidia's audio model), traditional CNNs, feature extraction (DINO v3), image generation (Flux), object detection, and video generation (Minimax H3 LTX).

  3. Multi-Device and Scaling Capabilities 25:50

    TRT MC supports multi-device setups, allowing large models to run in parallel across multiple GPUs (e.g., two or four Jetson/DGX Spark units) for accelerated inference, which is crucial for large-scale production systems.

  4. Full-Duplex and Complex Inference 19:10

    The platform supports complex, low-latency use cases, such as full-duplex voice chat (Nemotron Voice), which eliminates the need for separate ASR, LLM, and TTS pipelines, and advanced image understanding tasks like depth mapping and point cloud generation.

Watch on YouTube Full article

Hugging Face Journal Club: Direct On-Policy Distillation thumbnail

· 33:25

Hugging Face Journal Club: Direct On-Policy Distillation

The discussion details a novel technique called Direct On-Policy Distillation for achieving weak-to-strong generalization in large language models. This method proposes an efficient alternative to expensive full Reinforcement Learning (RL) training by leveraging the policy shift observed when training a small model with RL. Specifically, it uses this policy shift as a dense reward signal to train and update a much larger target model (student), significantly reducing computational costs compared to direct RL on the large model.

Key takeaways

  1. Weak-to-Strong Generalization via Policy Shift

    Instead of directly training a large model with expensive RL, this method measures how an RL run changes a small model's policy (the 'policy shift'). This shift is then used as a dense reward signal to distill knowledge into the larger target model.

  2. Efficiency Gains

    The technique offers substantial cost savings. For example, training a 7B model via RL might take 320 hours, while using distillation from a 1.5B model's policy shift can reduce the estimated time to around 164 hours.

  3. Methodological Blurring

    The process blurs the line between traditional RL and knowledge distillation by combining two types of losses: the policy shift signal (from RL) and a standard KL term, making the overall training setup highly efficient.

Watch on YouTube Full article

Local Models: Trust, Control, Optimization — Carter Abdallah, NVIDIA thumbnail

· 43:21

Local Models: Trust, Control, Optimization — Carter Abdallah, NVIDIA

The panel emphasized that for AI systems to achieve true sovereignty and trust, the ecosystem must be open—encompassing not just models but the entire training stack. Open weights allow users to own their data traces and customize models (e.g., Neotron, Trinity) via post-training environments, enabling specialized performance far exceeding generalized frontier closed APIs. The future points toward local/on-device compute becoming viable for most daily tasks, shifting AI development from relying solely on massive cloud endpoints.

Key takeaways

  1. Open Models Ensure Trust and Sovereignty 17:32

    Trust in open models is derived from verifiability: users can inspect the files, matrices, and running code (e.g., implementations from Prime Intellect, VLM, SGLang) rather than relying on unverifiable closed APIs. The ability to run a model locally ensures predictable output regardless of geopolitical or corporate access changes.

  2. Specialization Outperforms Generalization 22:00

    Open models allow for deep customization and post-training on specific use cases (e.g., finance automation). This specialization can yield better performance than generalized frontier models while being significantly cheaper to operate, enabling a data flywheel by allowing users to own their output traces.

  3. Local Compute is the Next Inflection Point 40:01

    The industry is moving toward local AI capability. The panel predicts that within the next year, open models will achieve capabilities comparable to frontier closed models (e.g., better than Fable), making it possible for most daily tasks to run on personal devices.

Watch on YouTube Full article

Hugging Face Journal Club: Scaling Laws for Pre-training & RL thumbnail

· 30:57

Hugging Face Journal Club: Scaling Laws for Pre-training & RL

The discussion analyzes a paper proposing a joint scaling law for pre-training and Reinforcement Learning (RL), which models how compute allocation across these stages impacts downstream task performance. Key findings suggest that increasing compute allocated to pre-training significantly boosts the model's ability on specific tasks (Pass 1) but has diminishing returns on generalized capability (Pass K). The analysis highlights the importance of optimizing the trade-off between SFT and RL compute budgets.

Key takeaways

  1. Pre-training vs. RL Compute Allocation 15:20

    The primary takeaway is that increasing compute allocated to pre-training leads to higher performance on downstream tasks (Pass 1). Conversely, while RL improves Pass 1, the model's generalized capability (Pass K) remains relatively stable regardless of the pre-training scale.

  2. Scaling Laws and Model Size 24:45

    When fixing the total compute budget, training smaller models for longer is generally more effective than attempting to train larger models, contradicting simple Chinchilla scaling assumptions in certain contexts.

Watch on YouTube Full article

Stanford CS329A Self-Improving AI Agents | Part 6 | Train Time Scaling/Scaling RL thumbnail

· 1:12:39

Stanford CS329A Self-Improving AI Agents | Part 6 | Train Time Scaling/Scaling RL

This lecture explores advanced techniques for scaling Large Language Model (LLM) reasoning capabilities through 'train-time scaling' and Reinforcement Learning (RL). The discussion covers three key papers: STaR (Self-Taught Reasoner), DeepSeekMath, and DAPO. Key findings demonstrate that smaller models can achieve high accuracy on complex benchmarks like AIME by leveraging structured training data generation (STaR) or advanced RL algorithms (DAPO/GRPO). The core insight is that closing the feedback loop—using model outputs to improve the model itself—is crucial for boosting reasoning, especially in domains with verifiability.

Key takeaways

  1. Train-Time Scaling vs. Test-Time Scaling 2:00

    While test-time scaling (inference-based techniques like majority voting) improves accuracy by sampling outputs, train-time scaling uses the model's own filtered outputs to fine-tune and improve the model weights directly, creating a powerful closed feedback loop.

  2. STaR Boosts Reasoning via Rationalization 6:30

    The STaR method bootstraps reasoning by generating solutions on a small set of examples. It filters for correct answers and then generates rationales (hints) for incorrect attempts, allowing the model to learn from failed paths iteratively.

  3. GRPO Addresses RL Memory Constraints 10:05

    DeepSeekMath introduced Group Relative Policy Optimization (GRPO), an efficient alternative to PPO that reduces memory overhead by using a group baseline instead of maintaining multiple policy copies, enabling scaling RL to larger models.

  4. DAPO Stabilizes Complex Reasoning 17:30

    DAPO addresses training instability in long chain-of-thought reasoning by implementing asymmetric clipping (allowing bigger increases) and dynamic sampling (filtering out zero or one reward groups to maintain a useful gradient signal).

Watch on YouTube Full article