Stanford CS229 Machine Learning | Spring 2026 | Lecture 6: Dataset Split, ML Advice
Summary
This lecture provides a deep dive into the fundamental challenge of machine learning: generalization. It systematically analyzes how models trained on finite, noisy samples can be selected to perform reliably on unseen data. Key concepts include decomposing test error into bias (model class limitation), variance (sensitivity to training data), and noise (measurement error). The discussion covers classical techniques like regularization (e.g., Ridge Regression) for reducing variance at the cost of slight bias, modern phenomena like Double Descent in overparameterized models, and practical model selection methods such as K-fold cross validation and Hyperband.
Key takeaways
-
Bias-Variance Decomposition
20:05
Test error is decomposed into three components: noise (intrinsic data error), squared bias (how far the average prediction is from the true function, dependent on model class), and variance (how much the prediction jumps around across different training sets). Minimizing test error requires balancing these three sources of error.
-
Regularization as Variance Reduction
30:05
Regularization techniques, such as adding an $L_2$ penalty (Ridge Regression), constrain the model weights ($ heta$) to prevent them from becoming too large. This stabilizes the model, significantly reducing variance while accepting a small increase in bias.
-
Modern ML Phenomena: Double Descent
1:03:25
The classical Bias-Variance curve suggests that test error must rise after a certain model complexity threshold. However, modern models can exhibit 'Double Descent,' where the test error decreases again in the massively overparameterized regime (i.e., having more parameters than data points).
-
Model Selection and Hyperparameter Tuning
50:50
To prevent information leakage from the test set, techniques like K-fold cross validation are used. For compute efficiency in tuning hyperparameters (e.g., regularization strength $ ho$), algorithms like Hyperband efficiently allocate computational resources to promising model configurations.
Technical details
-
Bias-Variance Decomposition Formula
1500s
The expected test error is decomposed into three terms: $ ext{Noise}^2$ (intrinsic data noise), $ ext{Bias}^2$, and $ ext{Variance}$. The variance term measures the fluctuation of the learned hypothesis $H_S$ across different training sets $S$.
-
Ridge Regression ($L_2$ Regularization)
1950s
The objective function is modified by adding an $L_2$ penalty term ($ ho || heta||^2$) to the standard least squares cost. This penalizes large weights, shrinking $ heta$ toward the origin and stabilizing the solution, which reduces variance.
-
Hyperband Algorithm
3050s
A compute-efficient method for tuning hyperparameters. It iteratively runs candidate models for a small number of steps (e.g., across different 'octaves' of $ ho$), dropping the worst half in each round and dedicating exponentially more resources to the most promising configurations.
-
Cross-Validation
4200s
A method for model selection that addresses the 'sacred' nature of the test set. Instead, the training data is split into multiple folds; the model is trained on $N-1$ folds and validated on the remaining fold, ensuring maximum data utilization.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.