Specializing AI for Regulated Industries - How Domyn Uses NVIDIA Nemotron
Summary
This livestream details Domyn's journey toward building a family of sovereign AI models for regulated industries, emphasizing full ownership and control over the entire model stack. The presentation covers advanced LLM development techniques—including model compression (pruning/distillation), continual pretraining (CPT), Supervised Fine-Tuning (SFT), and Reinforcement Learning (RL)—using the NVIDIA open source ecosystem. A significant focus is placed on the tooling required to manage these complex pipelines at scale, including custom tools like Swarm and FinalFlows, which are crucial for build engineers managing large-scale compute infrastructure.
Key takeaways
-
Sovereign AI requires full stack control
0:28
For regulated industries, achieving sovereign AI necessitates controlling every layer of the stack—from compute to model ownership—rather than relying on point solutions. This approach is critical for governance and auditability. (0:28-1:35)
-
Domain Large development pipeline
7:16
Domyn developed Domain Large by starting with Coliseum 355, followed by model compression (pruning/distillation), CPT to expand context up to 128K tokens, and SFT to enable reasoning. This was executed using the NVIDIA stack on H200s in DJX cloud. (4:36-7:39)
-
Domain Small for efficiency
23:43
To address cost concerns associated with large models, Domain Small (10B parameters) was created. Its training heavily leveraged Reinforcement Learning (RL) and Direct Preference Optimization (DPO), demonstrating that smaller models can achieve strong performance through advanced post-training curricula. (14:23-17:56)
-
Tooling for scalable ML pipelines
40:08
Domyn developed internal tools like Swarm (CLI/Python) and FinalFlows (DAG library) to manage complex, interconnected jobs on Slurm clusters. These tools remove friction when running large-scale evaluations and training across European infrastructure. (24:08-31:56)
Technical details
-
Model Compression & Optimization
638s
The process involved pruning and distillation, with the finding that cutting only ~10% of depth/width was sufficient to recover almost all model performance. The loss function used included cross-entropy regional one and KL loss acting as a regularizer. (6:38-7:29)
-
Training Methodologies
739s
The development utilized advanced techniques including Continual Pretraining (CPT) up to 128K tokens, Supervised Fine-Tuning (SFT), and Reinforcement Learning with verifiable rewards. DPO was used for post-training optimization on Domain Small. (7:39-10:45; 16:00-17:56)
-
Agentic Capabilities & Tool Use
1206s
The platform allows models to act as agentic systems, demonstrated by Domain Large generating complex SQL queries based on a provided schema. The architecture supports integrating external tools (e.g., web search) into the model workflow. (12:06-13:45; 29:28-30:47)
-
Compute Infrastructure
759s
The models were trained on dozens of H200s in DJX cloud, and the final frontier model development is leveraging European compute clusters designed for scientific computing. (7:59; 38:40-39:30)
-
MLOps Tooling
2408s
Domyn developed Swarm (CLI/Python) to manage inference replicas and load balancing on Slurm, Domino for aggregating evaluations across multiple benchmark suites, and FinalFlows (a DAG library) for managing interconnected jobs with built-in fault tolerance. (24:08-31:56)
Mentioned resources
- NVIDIA Nemotron
- Domain Large
- Domain Small
- Megatron LM / Megatron Bridge
- Azure Foundry
- Hugging Face
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.