The Base Model Is Dead — Varun Singh, Arcee AI
Summary
The traditional paradigm of viewing a base language model as merely a reflection of the entire human web is obsolete. As LLMs advance toward complex reasoning and agentic behavior (e.g., interacting with software environments), the training focus shifts from raw web text to incorporating instruction data, synthetic reasoning traces, and post-training techniques earlier into the pre-training phase. This requires careful management of data mixes, load balancing coefficients (especially in MOEs), and establishing stable representations early on.
Key takeaways
-
The Decline of Raw Web Text
2:10
Historically, models like GPT-3 relied heavily on raw web scrapes (e.g., Common Crawl/WebText-2), which constituted up to 85% of the training data. Modern recipes show a significant decrease in web text's proportion, indicating that its value is diminishing relative to code and structured reasoning abilities.
-
Shift from Knowledge Prior to Capability Prior
8:00
The base model's role is changing from accumulating general world knowledge (the 'prior') to carrying the necessary prior for complex Reinforcement Learning (RL) tasks. RL is no longer a mere 'cherry on top,' but a core component that requires the base model to be prepared for advanced composition and reasoning.
-
Synthetic Data Integration
11:20
A key trend involves pulling post-training data (like SFT/Q&A chat data) and large-scale synthetic data back into the pre-training phase. This allows models to learn task representations, conversation shapes, and atomic skills from the very beginning.
Technical details
-
Model Training Paradigms
The current understanding suggests a shift away from viewing LLM training as strictly Pre-training $\rightarrow$ Mid-training $\rightarrow$ Post-training/RL. Instead, the process is better understood through two broad paradigms: Supervised Learning (SL) to next token prediction and RL.
-
Data Mix Recipes
750s
Open recipes like Nemotron and Kimi K2 demonstrate a trend of incorporating SFT-prefixed data into the pre-training mix, alongside increasing use of code datasets (e.g., Swallow Code).
-
Mixture of Experts (MOE) Stability
1050s
When training MOEs, a critical pitfall is ensuring stable representations early on. The data distribution seen in post-training must not drastically differ from pre-training to avoid massive imbalances and maintain proper load balancing coefficients.
-
Agentic Training
390s
Advanced training now involves exposing models to agentic traces—data that simulates interaction with software environments, allowing the model to learn how to build applications and perform useful work through RL.
Mentioned resources
- GPT-3 paper
- GLM 4.5 / GLM 5
- Nemotron 3 Ultra
- Kimi K2
- MEI Thinking 1 paper
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.