Stanford CS229 Machine Learning | Spring 2026 | Lecture 2: Supervised Learning Setup
Summary
This lecture establishes the foundational mathematical framework for supervised machine learning, focusing on linear regression as a canonical example. Key concepts include defining a hypothesis function ($h: X o Y$), minimizing error using $ ext{L}_2$ loss (Mean Squared Error), and implementing optimization via Gradient Descent. The discussion emphasizes that modern ML success relies heavily on scaling models and computational efficiency, particularly through techniques like mini-batching and understanding the relationship between batch size and generalization.
Key takeaways
-
Supervised Learning Setup
17:12
Supervised learning requires a training set of paired examples $(X_i, Y_i)$, where $X$ is the input feature space (e.g., images, house data) and $Y$ is the corresponding label or value. The goal is to learn a hypothesis function $h$ that generalizes well to unseen data.
-
Regression vs. Classification
22:00
If the output $Y$ is a continuous real number (e.g., house price), it is a regression problem. If $Y$ is discrete or categorical (e.g., cat/dog label, positive/negative review), it is a classification problem.
-
Optimization via $ ext{L}_2$ Loss
30:40
The model parameters ($ heta$) are chosen by minimizing the empirical risk, typically using the squared error (the $ ext{L}_2$ norm). The objective function $J( heta)$ is minimized to find the best fit for all training points.
-
Stochastic Gradient Descent (SGD)
59:50
To handle massive datasets, optimization uses SGD, which updates parameters by calculating gradients over small subsets of data called mini-batches. This approach is computationally efficient and critical for scaling large models.
Technical details
-
Hypothesis Function
2350s
A hypothesis $h$ is defined as a function from an abstract set $X$ to an abstract set $Y$. For linear models, the function is parameterized by weights $ heta$: $h(x) = heta_0 + heta_1 x_1 + ... + heta_d x_d$. The use of $x_0=1$ simplifies notation and allows for a consistent form across dimensions.
-
Linear Regression Model
3050s
The linear model is considered the 'workhorse' algorithm. The goal is to find parameters $ heta$ that minimize the loss function $J( heta) = rac{1}{2N} ext{sum}_{i=1}^{N} (h_ heta(x_i) - y_i)^2$. This minimization leads to the Normal Equations solution: $ heta = (X^T X)^{-1} X^T Y$, assuming $X^T X$ is invertible.
-
Gradient Descent Mechanics
3200s
Optimization involves iteratively updating parameters by moving in the direction of maximal decrease (the negative gradient). The update rule uses a step size ($ ext{alpha}$ or learning rate) and is calculated based on the average error over the current mini-batch.
-
Mini-Batching
3590s
Instead of calculating gradients over the entire dataset (slow for large $N$), SGD uses random sampling from a small batch size ($B$). The update is based on averaging the error only over these $B$ samples, which significantly improves computational throughput.
Mentioned resources
- CS229 Machine Learning Course Notes
- Stanford TA Sessions
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.