Build A Reasoning Model From Scratch 4: Inference Scaling 1 (Temperature, Top-p, Self-Consistency)
This session details advanced inference-time scaling techniques used to improve Large Language Model (LLM) accuracy without retraining the base model. Techniques covered include Chain-of-Thought (CoT) prompting, Temperature Scaling, Top-p Filtering, and Self-Consistency (majority voting). The core mechanism involves modifying the text generation function to support diverse sampling, demonstrating that while these methods significantly boost accuracy (e.g., achieving 52% accuracy on Math500), they also dramatically increase computational cost and latency.
Key takeaways
-
Inference Scaling vs. Training Scaling
Improving model performance can be achieved by increasing compute either during training (e.g., larger models, more data) or during inference (inference-time scaling). Inference scaling techniques include generating more tokens or using advanced sampling methods like majority voting.
-
Chain-of-Thought (CoT) Prompting
29:03
Adding prompts like 'Let's think step by step' is a simple, effective modification that significantly boosts the base model's accuracy by forcing intermediate reasoning steps.
-
Temperature Scaling and Sampling
1:03:40
Temperature controls the output diversity by rescaling logits before applying softmax. A lower temperature (e.g., 0.5) increases the peak probability, making the model more deterministic, while a higher temperature increases randomness.
-
Top-p Filtering
This technique improves quality by filtering out low-probability tokens, keeping only the most likely tokens whose cumulative probability sum up to a threshold $p$ (e.g., 0.8).
-
Self-Consistency
Self-consistency uses majority voting: the LLM generates multiple answers (e.g., 3 or 5) using diverse sampling, and the most frequent answer is selected as the final, most robust result. This method is highly effective for numerical tasks like Math500.