# How are large language models trained?

## Executive summary

Large Language Models (LLMs) undergo two core training phases: pre-training and post-training. Pre-training uses next token prediction on massive datasets to build foundational language knowledge. Post-training refines this model for safety, helpfulness, and instruction following using techniques like Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). The process requires complex engineering at scale, including managing petabytes of data and utilizing advanced evaluation methods like auto-raters.

## Key takeaways

- Pre-training: Foundational Knowledge: The initial phase trains the model using next token prediction (a self-supervised task). This allows the LLM to implicitly learn language patterns and world knowledge without manual labeling. Scaling this process is a massive engineering challenge, requiring complex parallelism across thousands of interconnected GPUs, automated recovery systems, and pipelines for petabytes of data.
- Post-training: Alignment and Refinement: This phase focuses on aligning the model to be safe, helpful, and accurate. It primarily involves Supervised Fine-Tuning (SFT) using gold standard examples, or Reinforcement Learning (RL), where a separate Reward Model grades responses for quality, pushing the LLM toward desired behaviors.
- Evaluation and Measurement: Measuring model success requires defining clear criteria. While pre-training measures next word prediction accuracy, post-training focuses on alignment (e.g., helpfulness). Evaluation is scaled using a second LLM—an auto rater—to score responses against defined criteria.

## Technical details

- Pre-training Mechanism: The model is trained on next token prediction, where it predicts the subsequent word in a sequence. This process utilizes self-supervised learning by chunking publicly available data into inputs and outputs.
- Scaling Challenges: Due to model size, pre-training requires complex engineering solutions: using parallelism across thousands of GPUs, implementing automated recovery systems for crashes, and managing massive pipelines for petabytes of data.
- Supervised Fine-Tuning (SFT): SFT trains the model on curated 'gold standard' interactions. The loss is measured between the predicted response and the ideal label, updating the model weights to match desired behavior.
- Reinforcement Learning (RL): In RL, the LLM generates responses that are scored by a separate Reward Model. The model updates its weights based on these scores, allowing it to surpass the limitations of the curated SFT data.
- Auto-Rating: For evaluation, an auto rater (a second LLM) is used to score a model's responses against predefined criteria for tasks like helpfulness or humor, allowing progress tracking without manual effort.

## Practical implications

- The combination of SFT (setting the floor for basic competence) and RL (raising the ceiling by exploring optimal responses) is critical for creating production-ready LLMs.
- Model evaluation requires defining clear criteria upfront; otherwise, measuring abstract qualities like 'helpfulness' or 'insightful' becomes impossible.

## Topics

Large Language Models, Machine Learning Training, Deep Learning Architecture, AI Scaling and Infrastructure, Google AI, Gemini

Source: https://www.youtube.com/watch?v=JRArFxEfyQU