How are large language models trained?
Summary
Large Language Models (LLMs) undergo two core training phases: pre-training and post-training. Pre-training uses next token prediction on massive datasets to build foundational language knowledge. Post-training refines this model for safety, helpfulness, and instruction following using techniques like Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). The process requires complex engineering at scale, including managing petabytes of data and utilizing advanced evaluation methods like auto-raters.
Key takeaways
-
Pre-training: Foundational Knowledge
The initial phase trains the model using next token prediction (a self-supervised task). This allows the LLM to implicitly learn language patterns and world knowledge without manual labeling. Scaling this process is a massive engineering challenge, requiring complex parallelism across thousands of interconnected GPUs, automated recovery systems, and pipelines for petabytes of data.
-
Post-training: Alignment and Refinement
5:40
This phase focuses on aligning the model to be safe, helpful, and accurate. It primarily involves Supervised Fine-Tuning (SFT) using gold standard examples, or Reinforcement Learning (RL), where a separate Reward Model grades responses for quality, pushing the LLM toward desired behaviors.
-
Evaluation and Measurement
7:40
Measuring model success requires defining clear criteria. While pre-training measures next word prediction accuracy, post-training focuses on alignment (e.g., helpfulness). Evaluation is scaled using a second LLM—an auto rater—to score responses against defined criteria.
Technical details
-
Pre-training Mechanism
105s
The model is trained on next token prediction, where it predicts the subsequent word in a sequence. This process utilizes self-supervised learning by chunking publicly available data into inputs and outputs.
-
Scaling Challenges
170s
Due to model size, pre-training requires complex engineering solutions: using parallelism across thousands of GPUs, implementing automated recovery systems for crashes, and managing massive pipelines for petabytes of data.
-
Supervised Fine-Tuning (SFT)
280s
SFT trains the model on curated 'gold standard' interactions. The loss is measured between the predicted response and the ideal label, updating the model weights to match desired behavior.
-
Reinforcement Learning (RL)
320s
In RL, the LLM generates responses that are scored by a separate Reward Model. The model updates its weights based on these scores, allowing it to surpass the limitations of the curated SFT data.
-
Auto-Rating
490s
For evaluation, an auto rater (a second LLM) is used to score a model's responses against predefined criteria for tasks like helpfulness or humor, allowing progress tracking without manual effort.
Mentioned resources
- Google AI
- Gemini
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.