# Stanford CS229 Machine Learning | Spring 2026 | Lecture 6: Dataset Split, ML Advice

## Executive summary

This lecture provides a deep dive into the fundamental challenge of machine learning: generalization. It systematically analyzes how models trained on finite, noisy samples can be selected to perform reliably on unseen data. Key concepts include decomposing test error into bias (model class limitation), variance (sensitivity to training data), and noise (measurement error). The discussion covers classical techniques like regularization (e.g., Ridge Regression) for reducing variance at the cost of slight bias, modern phenomena like Double Descent in overparameterized models, and practical model selection methods such as K-fold cross validation and Hyperband.

## Key takeaways

- Bias-Variance Decomposition: Test error is decomposed into three components: noise (intrinsic data error), squared bias (how far the average prediction is from the true function, dependent on model class), and variance (how much the prediction jumps around across different training sets). Minimizing test error requires balancing these three sources of error.
- Regularization as Variance Reduction: Regularization techniques, such as adding an $L_2$ penalty (Ridge Regression), constrain the model weights ($ heta$) to prevent them from becoming too large. This stabilizes the model, significantly reducing variance while accepting a small increase in bias.
- Modern ML Phenomena: Double Descent: The classical Bias-Variance curve suggests that test error must rise after a certain model complexity threshold. However, modern models can exhibit 'Double Descent,' where the test error decreases again in the massively overparameterized regime (i.e., having more parameters than data points).
- Model Selection and Hyperparameter Tuning: To prevent information leakage from the test set, techniques like K-fold cross validation are used. For compute efficiency in tuning hyperparameters (e.g., regularization strength $ ho$), algorithms like Hyperband efficiently allocate computational resources to promising model configurations.

## Technical details

- Bias-Variance Decomposition Formula: The expected test error is decomposed into three terms: $ ext{Noise}^2$ (intrinsic data noise), $ ext{Bias}^2$, and $ ext{Variance}$. The variance term measures the fluctuation of the learned hypothesis $H_S$ across different training sets $S$.
- Ridge Regression ($L_2$ Regularization): The objective function is modified by adding an $L_2$ penalty term ($ ho || heta||^2$) to the standard least squares cost. This penalizes large weights, shrinking $ heta$ toward the origin and stabilizing the solution, which reduces variance.
- Hyperband Algorithm: A compute-efficient method for tuning hyperparameters. It iteratively runs candidate models for a small number of steps (e.g., across different 'octaves' of $ ho$), dropping the worst half in each round and dedicating exponentially more resources to the most promising configurations.
- Cross-Validation: A method for model selection that addresses the 'sacred' nature of the test set. Instead, the training data is split into multiple folds; the model is trained on $N-1$ folds and validated on the remaining fold, ensuring maximum data utilization.

## Practical implications

- When designing ML systems, always distinguish between training data (used for fitting) and test/validation data (used only for final evaluation).
- Use techniques like cross-validation or hold-out sets to tune hyperparameters ($ ho$) without leaking information from the ultimate test set.
- Understand that model robustness can be achieved by intentionally adding regularization, which constrains complexity and prevents overfitting to noise.

## Topics

Bias-Variance Tradeoff, Regularization (L1/L2), Model Selection, Cross-Validation, Hyperparameter Optimization, Double Descent, Overfitting and Underfitting, Stanford CS229 Machine Learning, General AI Information

Source: https://www.youtube.com/watch?v=llnEgyyuYkQ