# Stanford CS229 Machine Learning | Spring 2026 | Lecture 2: Supervised Learning Setup

## Executive summary

This lecture establishes the foundational mathematical framework for supervised machine learning, focusing on linear regression as a canonical example. Key concepts include defining a hypothesis function ($h: X o Y$), minimizing error using $ ext{L}_2$ loss (Mean Squared Error), and implementing optimization via Gradient Descent. The discussion emphasizes that modern ML success relies heavily on scaling models and computational efficiency, particularly through techniques like mini-batching and understanding the relationship between batch size and generalization.

## Key takeaways

- Supervised Learning Setup: Supervised learning requires a training set of paired examples $(X_i, Y_i)$, where $X$ is the input feature space (e.g., images, house data) and $Y$ is the corresponding label or value. The goal is to learn a hypothesis function $h$ that generalizes well to unseen data.
- Regression vs. Classification: If the output $Y$ is a continuous real number (e.g., house price), it is a regression problem. If $Y$ is discrete or categorical (e.g., cat/dog label, positive/negative review), it is a classification problem.
- Optimization via $ ext{L}_2$ Loss: The model parameters ($ heta$) are chosen by minimizing the empirical risk, typically using the squared error (the $ ext{L}_2$ norm). The objective function $J( heta)$ is minimized to find the best fit for all training points.
- Stochastic Gradient Descent (SGD): To handle massive datasets, optimization uses SGD, which updates parameters by calculating gradients over small subsets of data called mini-batches. This approach is computationally efficient and critical for scaling large models.

## Technical details

- Hypothesis Function: A hypothesis $h$ is defined as a function from an abstract set $X$ to an abstract set $Y$. For linear models, the function is parameterized by weights $ heta$: $h(x) = heta_0 + heta_1 x_1 + ... + heta_d x_d$. The use of $x_0=1$ simplifies notation and allows for a consistent form across dimensions.
- Linear Regression Model: The linear model is considered the 'workhorse' algorithm. The goal is to find parameters $ heta$ that minimize the loss function $J( heta) = rac{1}{2N} ext{sum}_{i=1}^{N} (h_ heta(x_i) - y_i)^2$. This minimization leads to the Normal Equations solution: $ heta = (X^T X)^{-1} X^T Y$, assuming $X^T X$ is invertible.
- Gradient Descent Mechanics: Optimization involves iteratively updating parameters by moving in the direction of maximal decrease (the negative gradient). The update rule uses a step size ($ ext{alpha}$ or learning rate) and is calculated based on the average error over the current mini-batch.
- Mini-Batching: Instead of calculating gradients over the entire dataset (slow for large $N$), SGD uses random sampling from a small batch size ($B$). The update is based on averaging the error only over these $B$ samples, which significantly improves computational throughput.

## Practical implications

- The principles of mini-batching and gradient descent are fundamental to training modern large language models (LLMs) and deep neural networks.
- Computational efficiency is paramount; the choice of batch size often becomes an engineering concern (e.g., GPU memory utilization) rather than purely a statistical one.
- Understanding model scaling laws and optimization constraints (like those related to GPU architecture) is crucial for building state-of-the-art AI systems.

## Topics

Supervised Learning, Linear Regression, Gradient Descent, Stochastic Gradient Descent (SGD), Loss Functions ($	ext{L}_2$), Mini-Batching, Machine Learning Theory, CS229 Machine Learning Course Notes, Stanford TA Sessions

Source: https://www.youtube.com/watch?v=cmNIMjPYdgM