# Ask the Experts: Inside Nemotron Post-Training | Nemotron Labs

## Executive summary

This session details the advanced post-training pipeline used by NVIDIA AI researchers to build state-of-the-art (SOTA) models like Nemotron, focusing on enhancing intelligence and enabling agentic capabilities. Post-training is presented as an evolution from traditional task-specific training, utilizing structured data, chat templates, and specialized frameworks like NeMo Gym and NeMo RL. Key strategies discussed include performing ablation studies on data subsets, employing Mixture of Experts (MoE) for capability generalization, and establishing robust feedback loops using real-world user data to prevent model degradation.

## Key takeaways

- Post-Training Evolution: Post-training builds upon pre-training (which uses massive, diverse, unstructured data for causal language modeling) by focusing on structured data. This teaches the model to follow instructions, use chat templates, and emit tool calls, moving beyond simple task-specific or multi-task training.
- Structured Data for Capabilities: To narrow model focus, post-training emphasizes structured data (e.g., tool responses, user prompts, tool sets) over unstructured text. This allows the model to learn specific formats, such as JSON or XML, for tool interaction.
- Mitigating Capability Degradation: To specialize a model (e.g., for software engineering) without losing general capabilities, techniques include including general chat data in the blend and using algorithmic approaches like multi-teacher on-policy distillation (MOPD).
- Data Bias and Profiling: Data quality is paramount. Researchers must analyze data statistics (e.g., trajectory length, tool call diversity) and perform ablation studies to identify and address biases (e.g., over-reliance on a single tool) that could cause model failure in real-world use.
- Starting the Pipeline: For developers, the process involves defining the model's 'soul' (core capabilities), quantifying these goals via benchmarks, setting quantitative milestones, and iterating through the post-training pipeline. Starting resources include Nemotron 3.5 Lightning.

## Technical details

- Post-Training Methodology: The process involves training a pre-trained model on structured data to teach specific behaviors like following chat templates and emitting tool calls. This is distinct from pre-training's causal language modeling objective on massive, diverse corpuses.
- Training Frameworks: NVIDIA provides specialized libraries for post-training workflows: NeMo Data Designer (for synthetic dataset creation), NeMo Gym (for building RL/EVO environments), and NeMo RL (for reinforcement learning training).
- Data Blending and Bias Correction: To ensure robust performance, data blends must be meaningfully diverse. Data profiling involves analyzing statistics (e.g., tool call correlation, trajectory length) to identify biases. Ablation studies are used to test the impact of specific data subsets in isolation.
- Advanced Optimization: Techniques like Multi-Teacher On-Policy Distillation (MOPD) are used to preserve general capabilities while optimizing for multiple specialized domains (e.g., search, software engineering).
- Feedback Loop Design: The feedback loop between real-world user data and model improvement is managed by analyzing the distribution of real user data and creating synthetic clean data to model this distribution, ensuring the model learns reliable patterns.

## Practical implications

- Build engineers should incorporate data profiling and ablation studies into the model training pipeline to quantify and mitigate data biases.
- The use of structured data (e.g., chat templates, tool call formats) is critical for moving models from general intelligence to reliable, task-specific agentic behavior.
- Pipelines should consider advanced optimization techniques like MOPD to balance specialization with general capability retention.
- The process requires a deep understanding of the model's 'soul' (core capabilities) to guide data curation and resource allocation.

## Topics

Large Language Models (LLMs), Reinforcement Learning (RL), Supervised Fine-Tuning (SFT), Data Engineering, Agentic AI, Model Optimization, NeMo Data Designer, NeMo Gym, NeMo RL, Nemotron GitHub, HuggingFace, Nemotron 3.5 Lightning

Source: https://www.youtube.com/watch?v=V60wEnedg4g