# Stanford CS229 Machine Learning | Spring 2026 | Lecture 8: Neural Networks 2 (Backprop)

## Executive summary

This lecture provides a deep theoretical dive into Backpropagation and Automatic Differentiation, establishing it as an efficient method for computing gradients in complex neural networks. The core principle is that any differentiable network can be viewed as a 'differentiable circuit' or computational graph. This allows the gradient (the backward pass) to be computed with a time complexity proportional to the number of parameters ($O(N)$), matching the efficiency of the forward pass, regardless of how complex the underlying function is.

## Key takeaways

- Automatic Differentiation and Computational Graphs: The gradient computation (backward pass) can be viewed as traversing a differentiable circuit. This method allows for efficient calculation because it reuses intermediate results, unlike expanding the function into a traditional mathematical formula.
- Efficiency of Gradient Computation: The fundamental theorem states that if a differentiable circuit of size $N$ computes a real-valued function, its gradient can also be computed in time complexity $O(N)$. This means the forward pass (evaluating loss) and the backward pass (calculating gradients) have similar computational costs.
- Chain Rule Application for Backprop: Backpropagation is an application of the Chain Rule. By knowing the gradient with respect to an intermediate variable ($U$), one can compute the gradient with respect to its input ($Z$) using matrix multiplication involving the Jacobian (or transpose of the Jacobian). This process allows computation to proceed layer by layer.

## Technical details

- Gradient for Matrix Multiplication: If a function $G$ is defined as matrix multiplication, $U = WZ + B$, the gradient with respect to the input $Z$ (given $ rac{ ext{d}J}{ ext{d}U}$) simplifies dramatically: $ rac{ ext{d}J}{ ext{d}Z} = W^T imes rac{ ext{d}J}{ ext{d}U}$. The backward function is simply the transpose of the forward matrix.
- Gradient for Parameter Weights (W): To compute $ rac{ ext{d}J}{ ext{d}W}$, one must apply the chain rule, resulting in a calculation that is equivalent to an outer product: $ rac{ ext{d}J}{ ext{d}W} = rac{ ext{d}J}{ ext{d}U} imes Z^T$. This gradient for a single example is always rank one.
- Backprop through Activation Functions: For an activation function $U = ext{Sigmoid}(Z)$, the derivative $ rac{ ext{d}J}{ ext{d}Z}$ is computed by element-wise multiplication: $ rac{ ext{d}J}{ ext{d}Z} = rac{ ext{d}J}{ ext{d}U} imes ext{Sigmoid}'(Z)$. This can be efficiently represented as a diagonal matrix multiplication.
- Second-Order Derivatives: The principles of automatic differentiation extend to second-order methods. For example, computing the gradient of a loss function after one step of gradient descent ($ rac{ ext{d}}{ ext{d} heta} ext{Loss}( heta - ext{step})$) remains efficiently computable because all intermediate steps are themselves differentiable circuits.

## Practical implications

- The understanding of computational graphs is foundational for optimizing deep learning frameworks (like PyTorch or TensorFlow), allowing them to manage memory and compute gradients efficiently.
- This theory enables advanced optimization techniques, such as calculating second-order derivatives (Hessian matrix approximations) needed for sophisticated optimizers.
- It allows complex algorithms—including meta-learning processes like tuning a learning rate via backpropagation through the algorithm itself—to be optimized using standard gradient descent methods.

## Topics

Automatic Differentiation, Backpropagation, Neural Networks, Computational Graphs, Chain Rule, Optimization Theory, Stanford CS229 Machine Learning, Stanford AI Programs Information

Source: https://www.youtube.com/watch?v=ne2ngVAoMG8