# Build A Reasoning Model From Scratch 4: Inference Scaling 1 (Temperature, Top-p, Self-Consistency)

## Executive summary

This session details advanced inference-time scaling techniques used to improve Large Language Model (LLM) accuracy without retraining the base model. Techniques covered include Chain-of-Thought (CoT) prompting, Temperature Scaling, Top-p Filtering, and Self-Consistency (majority voting). The core mechanism involves modifying the text generation function to support diverse sampling, demonstrating that while these methods significantly boost accuracy (e.g., achieving 52% accuracy on Math500), they also dramatically increase computational cost and latency.

## Key takeaways

- Inference Scaling vs. Training Scaling: Improving model performance can be achieved by increasing compute either during training (e.g., larger models, more data) or during inference (inference-time scaling). Inference scaling techniques include generating more tokens or using advanced sampling methods like majority voting.
- Chain-of-Thought (CoT) Prompting: Adding prompts like 'Let's think step by step' is a simple, effective modification that significantly boosts the base model's accuracy by forcing intermediate reasoning steps.
- Temperature Scaling and Sampling: Temperature controls the output diversity by rescaling logits before applying softmax. A lower temperature (e.g., 0.5) increases the peak probability, making the model more deterministic, while a higher temperature increases randomness.
- Top-p Filtering: This technique improves quality by filtering out low-probability tokens, keeping only the most likely tokens whose cumulative probability sum up to a threshold $p$ (e.g., 0.8).
- Self-Consistency: Self-consistency uses majority voting: the LLM generates multiple answers (e.g., 3 or 5) using diverse sampling, and the most frequent answer is selected as the final, most robust result. This method is highly effective for numerical tasks like Math500.

## Technical details

- Inference-Time Scaling: This refers to methods used when running the LLM after training to improve performance, such as generating more tokens or using advanced sampling techniques.
- Text Generation Modification: The standard text generation function is modified to accept parameters for temperature and Top-p filtering, allowing the implementation of advanced sampling methods.
- Temperature Scaling Mechanism: The logits ($z$) are rescaled by dividing by the temperature ($ au$): $z' = z / au$. This influences the subsequent probability distribution via softmax.
- Top-p Filtering: The process involves sorting the token probabilities in descending order, calculating the cumulative sum, and masking out all tokens whose cumulative probability exceeds the specified threshold $p$. The remaining tokens are then renormalized.
- Self-Consistency: The process involves running the LLM multiple times (e.g., 3 or 5) with diverse sampling (using temperature and Top-p) and selecting the most frequently occurring answer (majority voting).

## Practical implications

- Self-consistency is highly effective for numerical tasks (like Math500) where majority voting can significantly boost accuracy.
- The choice of inference technique involves a trade-off: increased accuracy comes at the cost of increased computational resources and latency.
- For production systems, self-consistency should only be used when accuracy is paramount and there are no strict latency constraints.
- Understanding temperature and Top-p is crucial for controlling the model's output diversity, allowing for controlled creativity or deterministic results.

## Topics

LLM Inference, Prompt Engineering, Sampling Techniques, Majority Voting, Computational Optimization, Reasoning playlist, Reasoning Book, Reasoning GitHub repo, LLMs from Scratch book, LLMs from Scratch playlist

Source: https://www.youtube.com/watch?v=t5y-kS9nNxU