Stanford CS229 Machine Learning | Spring 2026 | Lecture 8: Neural Networks 2 (Backprop)
Summary
This lecture provides a deep theoretical dive into Backpropagation and Automatic Differentiation, establishing it as an efficient method for computing gradients in complex neural networks. The core principle is that any differentiable network can be viewed as a 'differentiable circuit' or computational graph. This allows the gradient (the backward pass) to be computed with a time complexity proportional to the number of parameters ($O(N)$), matching the efficiency of the forward pass, regardless of how complex the underlying function is.
Key takeaways
-
Automatic Differentiation and Computational Graphs
2:00
The gradient computation (backward pass) can be viewed as traversing a differentiable circuit. This method allows for efficient calculation because it reuses intermediate results, unlike expanding the function into a traditional mathematical formula.
-
Efficiency of Gradient Computation
2:40
The fundamental theorem states that if a differentiable circuit of size $N$ computes a real-valued function, its gradient can also be computed in time complexity $O(N)$. This means the forward pass (evaluating loss) and the backward pass (calculating gradients) have similar computational costs.
-
Chain Rule Application for Backprop
5:50
Backpropagation is an application of the Chain Rule. By knowing the gradient with respect to an intermediate variable ($U$), one can compute the gradient with respect to its input ($Z$) using matrix multiplication involving the Jacobian (or transpose of the Jacobian). This process allows computation to proceed layer by layer.
Technical details
-
Gradient for Matrix Multiplication
570s
If a function $G$ is defined as matrix multiplication, $U = WZ + B$, the gradient with respect to the input $Z$ (given $ rac{ ext{d}J}{ ext{d}U}$) simplifies dramatically: $ rac{ ext{d}J}{ ext{d}Z} = W^T imes rac{ ext{d}J}{ ext{d}U}$. The backward function is simply the transpose of the forward matrix.
-
Gradient for Parameter Weights (W)
680s
To compute $ rac{ ext{d}J}{ ext{d}W}$, one must apply the chain rule, resulting in a calculation that is equivalent to an outer product: $ rac{ ext{d}J}{ ext{d}W} = rac{ ext{d}J}{ ext{d}U} imes Z^T$. This gradient for a single example is always rank one.
-
Backprop through Activation Functions
850s
For an activation function $U = ext{Sigmoid}(Z)$, the derivative $ rac{ ext{d}J}{ ext{d}Z}$ is computed by element-wise multiplication: $ rac{ ext{d}J}{ ext{d}Z} = rac{ ext{d}J}{ ext{d}U} imes ext{Sigmoid}'(Z)$. This can be efficiently represented as a diagonal matrix multiplication.
-
Second-Order Derivatives
300s
The principles of automatic differentiation extend to second-order methods. For example, computing the gradient of a loss function after one step of gradient descent ($ rac{ ext{d}}{ ext{d} heta} ext{Loss}( heta - ext{step})$) remains efficiently computable because all intermediate steps are themselves differentiable circuits.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.