Back Propagation

Overview

Backpropagation is a method for efficiently computing the gradient of the loss function with respect to the parameters of a neural network. It relies on the chain rule from calculus to propagate information about the loss backward through the network.

In any neural network, calculus does a lot of the heavy lifting. Imagine we're training a neural network, and backpropagation has not yet been baked into every library everywhere. When we get to finding gradients, we'll need to compute the derivative of the loss function with respect to many parameters. A naïve approach could repeatedly recompute the same intermediate derivatives, quickly becoming inefficient for a large network.

The key insight behind backpropagation is that these intermediate derivatives can be reused. Instead of independently computing the effect of every parameter on the final loss, we work backward through the network and calculate each necessary intermediate gradient once.

The Chain Rule

Consider a simple composition of functions:

$$y=f(g(x)).$$

The chain rule tells us that

$$\frac{dy}{dx} = \frac{dy}{dg} \frac{dg}{dx}.$$

A neural network is essentially a much larger composition of functions. Each layer takes the output of the previous layer, applies some transformation and activation function, and passes the result onward.

If the final output determines a loss $L$, we can use the chain rule to work backward from $L$ and determine how much each intermediate value contributed to the final error.

A Simple Network

Suppose we have a network with

$$z=wx+b$$

followed by an activation function

$$a=\sigma(z),$$

and finally a loss function $L(a,y)$ comparing the prediction $a$ with the target $y$.

To find the gradient with respect to the weight $w$, we can apply the chain rule:

$$\frac{\partial L}{\partial w} = \frac{\partial L}{\partial a} \frac{\partial a}{\partial z} \frac{\partial z}{\partial w}.$$

Each term describes one local step through the network. The same idea extends to much larger networks, where the output of one layer becomes the input to another.

The Algorithm

Roughly, training a neural network using backpropagation consists of a forward pass followed by a backward pass.

# Forward pass
z1 = W1 @ x + b1
a1 = activation(z1)

z2 = W2 @ a1 + b2
a2 = activation(z2)

loss = L(a2, y)

# Backward pass
dL_da2 = derivative_of_loss(a2, y)
dL_dz2 = dL_da2 * activation_derivative(z2)

dL_dW2 = dL_dz2 @ a1.T
dL_da1 = W2.T @ dL_dz2

dL_dz1 = dL_da1 * activation_derivative(z1)
dL_dW1 = dL_dz1 @ x.T
  

The forward pass computes the network's prediction and the resulting loss. The backward pass then uses the chain rule to propagate gradients from the loss back toward the input.

Once these gradients have been computed, an optimization algorithm such as gradient descent can update the parameters:

$$w\leftarrow w-\eta\frac{\partial L}{\partial w},$$

where $\eta$ is the learning rate.

Why Is It Efficient?

The important advantage of backpropagation is that intermediate quantities are reused. When the gradient flowing through one layer has already been computed, it can be used to determine the gradients of the parameters that produced that layer's output.

Instead of independently differentiating the entire loss with respect to every parameter, backpropagation traverses the computational graph and performs a constant amount of local work for each connection and operation. As a result, the cost of computing all gradients is generally proportional to the size of the computational graph.

For a network whose number of parameters and computational operations scale together, this makes gradient computation roughly linear in the size of the network rather than requiring a separate computation for every parameter.

A Broader Connection

Backpropagation is a particularly elegant application of a familiar mathematical idea. The chain rule itself is simple, but applying it systematically to a large computational graph allows us to efficiently compute gradients for models containing millions or even billions of parameters.

The algorithm is therefore a useful example of how an important algorithmic improvement can come not from changing the underlying mathematical objective, but from recognizing and exploiting repeated structure in the computation.