Stanford CS229 Machine Learning | Spring 2026 | Lecture 8: Neural Networks 2 (Backprop)
This lecture provides a deep theoretical dive into Backpropagation and Automatic Differentiation, establishing it as an efficient method for computing gradients in complex neural networks. The core principle is that any differentiable network can be viewed as a 'differentiable circuit' or computational graph. This allows the gradient (the backward pass) to be computed with a time complexity proportional to the number of parameters ($O(N)$), matching the efficiency of the forward pass, regardless of how complex the underlying function is.
Key takeaways
-
Automatic Differentiation and Computational Graphs
2:00
The gradient computation (backward pass) can be viewed as traversing a differentiable circuit. This method allows for efficient calculation because it reuses intermediate results, unlike expanding the function into a traditional mathematical formula.
-
Efficiency of Gradient Computation
2:40
The fundamental theorem states that if a differentiable circuit of size $N$ computes a real-valued function, its gradient can also be computed in time complexity $O(N)$. This means the forward pass (evaluating loss) and the backward pass (calculating gradients) have similar computational costs.
-
Chain Rule Application for Backprop
5:50
Backpropagation is an application of the Chain Rule. By knowing the gradient with respect to an intermediate variable ($U$), one can compute the gradient with respect to its input ($Z$) using matrix multiplication involving the Jacobian (or transpose of the Jacobian). This process allows computation to proceed layer by layer.