Stanford CS229 Machine Learning | Spring 2026 | Lecture 6: Dataset Split, ML Advice
This lecture provides a deep dive into the fundamental challenge of machine learning: generalization. It systematically analyzes how models trained on finite, noisy samples can be selected to perform reliably on unseen data. Key concepts include decomposing test error into bias (model class limitation), variance (sensitivity to training data), and noise (measurement error). The discussion covers classical techniques like regularization (e.g., Ridge Regression) for reducing variance at the cost of slight bias, modern phenomena like Double Descent in overparameterized models, and practical model selection methods such as K-fold cross validation and Hyperband.
Key takeaways
-
Bias-Variance Decomposition
20:05
Test error is decomposed into three components: noise (intrinsic data error), squared bias (how far the average prediction is from the true function, dependent on model class), and variance (how much the prediction jumps around across different training sets). Minimizing test error requires balancing these three sources of error.
-
Regularization as Variance Reduction
30:05
Regularization techniques, such as adding an $L_2$ penalty (Ridge Regression), constrain the model weights ($ heta$) to prevent them from becoming too large. This stabilizes the model, significantly reducing variance while accepting a small increase in bias.
-
Modern ML Phenomena: Double Descent
1:03:25
The classical Bias-Variance curve suggests that test error must rise after a certain model complexity threshold. However, modern models can exhibit 'Double Descent,' where the test error decreases again in the massively overparameterized regime (i.e., having more parameters than data points).
-
Model Selection and Hyperparameter Tuning
50:50
To prevent information leakage from the test set, techniques like K-fold cross validation are used. For compute efficiency in tuning hyperparameters (e.g., regularization strength $ ho$), algorithms like Hyperband efficiently allocate computational resources to promising model configurations.
