The Base Model Is Dead — Varun Singh, Arcee AI
The traditional paradigm of viewing a base language model as merely a reflection of the entire human web is obsolete. As LLMs advance toward complex reasoning and agentic behavior (e.g., interacting with software environments), the training focus shifts from raw web text to incorporating instruction data, synthetic reasoning traces, and post-training techniques earlier into the pre-training phase. This requires careful management of data mixes, load balancing coefficients (especially in MOEs), and establishing stable representations early on.
Key takeaways
-
The Decline of Raw Web Text
2:10
Historically, models like GPT-3 relied heavily on raw web scrapes (e.g., Common Crawl/WebText-2), which constituted up to 85% of the training data. Modern recipes show a significant decrease in web text's proportion, indicating that its value is diminishing relative to code and structured reasoning abilities.
-
Shift from Knowledge Prior to Capability Prior
8:00
The base model's role is changing from accumulating general world knowledge (the 'prior') to carrying the necessary prior for complex Reinforcement Learning (RL) tasks. RL is no longer a mere 'cherry on top,' but a core component that requires the base model to be prepared for advanced composition and reasoning.
-
Synthetic Data Integration
11:20
A key trend involves pulling post-training data (like SFT/Q&A chat data) and large-scale synthetic data back into the pre-training phase. This allows models to learn task representations, conversation shapes, and atomic skills from the very beginning.