# The Base Model Is Dead — Varun Singh, Arcee AI

## Executive summary

The traditional paradigm of viewing a base language model as merely a reflection of the entire human web is obsolete. As LLMs advance toward complex reasoning and agentic behavior (e.g., interacting with software environments), the training focus shifts from raw web text to incorporating instruction data, synthetic reasoning traces, and post-training techniques earlier into the pre-training phase. This requires careful management of data mixes, load balancing coefficients (especially in MOEs), and establishing stable representations early on.

## Key takeaways

- The Decline of Raw Web Text: Historically, models like GPT-3 relied heavily on raw web scrapes (e.g., Common Crawl/WebText-2), which constituted up to 85% of the training data. Modern recipes show a significant decrease in web text's proportion, indicating that its value is diminishing relative to code and structured reasoning abilities.
- Shift from Knowledge Prior to Capability Prior: The base model's role is changing from accumulating general world knowledge (the 'prior') to carrying the necessary prior for complex Reinforcement Learning (RL) tasks. RL is no longer a mere 'cherry on top,' but a core component that requires the base model to be prepared for advanced composition and reasoning.
- Synthetic Data Integration: A key trend involves pulling post-training data (like SFT/Q&A chat data) and large-scale synthetic data back into the pre-training phase. This allows models to learn task representations, conversation shapes, and atomic skills from the very beginning.

## Technical details

- Model Training Paradigms: The current understanding suggests a shift away from viewing LLM training as strictly Pre-training $\rightarrow$ Mid-training $\rightarrow$ Post-training/RL. Instead, the process is better understood through two broad paradigms: Supervised Learning (SL) to next token prediction and RL.
- Data Mix Recipes: Open recipes like Nemotron and Kimi K2 demonstrate a trend of incorporating SFT-prefixed data into the pre-training mix, alongside increasing use of code datasets (e.g., Swallow Code).
- Mixture of Experts (MOE) Stability: When training MOEs, a critical pitfall is ensuring stable representations early on. The data distribution seen in post-training must not drastically differ from pre-training to avoid massive imbalances and maintain proper load balancing coefficients.
- Agentic Training: Advanced training now involves exposing models to agentic traces—data that simulates interaction with software environments, allowing the model to learn how to build applications and perform useful work through RL.

## Practical implications

- Data pipelines must evolve to integrate structured and synthetic data (SFT/Q&A) into the initial pre-training phase.
- Model development requires careful management of training stages, particularly ensuring that post-training objectives are reflected in early base model representations to maintain stability.
- Compute allocation for RL is becoming increasingly dominant, requiring a re-evaluation of how supervised learning contributes to building useful representations for agentic tasks.

## Topics

LLM Training, Data Engineering, AI Architecture, Reinforcement Learning, GPT-3 paper, GLM 4.5 / GLM 5, Nemotron 3 Ultra, Kimi K2, MEI Thinking 1 paper

Source: https://www.youtube.com/watch?v=xbPriQWXtWM