Topic · Analysis

Backpropagation

One algorithm, calculus walked backwards, every neural network you've ever heard of.

Show a network a picture; it guesses. Measure how wrong the guess was; that number is the loss. Backprop walks the chain rule backwards through every layer to ask, for each one of the millions of weights, the same question: if I had nudged you up a tiny bit, would the loss have gone up or down? Then it nudges every weight in the direction that lowers the loss. Repeat a billion times. That single trick is GPT, AlphaFold, image generators, speech recognition — the entire modern AI boom on one mathematical idea.

Discover the story → Open the Explorer

First encounter

Calculus, walked backwards through a network.

The big idea

A neural network is a giant differentiable function from input to output, with millions of dials (weights) inside. Pick any output mistake, and the chain rule tells you exactly which dials to turn, in which direction, by how much. Backpropagation is that calculation, implemented as one efficient pass from the output back to the input. Every weight learns from every example, simultaneously.

A concrete example

You want a net that outputs 1 for the XOR of two bits and 0 otherwise. Start with random weights — the net guesses garbage. Compare guess to truth, get a loss. Backprop tells each weight: 'lower yourself by 0.03', 'raise yourself by 0.11', and so on. After a few hundred such mini-corrections, the same network outputs 0.98 instead of 0.4, and 0.02 instead of 0.6. It has learned the rule no single neuron could ever encode alone.

Why it matters

Backprop is almost embarrassingly simple — it is high-school chain rule plus the discipline to evaluate it once, layer by layer. But every modern AI system is the same algorithm at a different scale. From 1986's tiny networks to 2026's trillion-parameter models, nothing about the update rule has changed. The hardware grew, the data grew, the network grew. The math stayed.

Train a tiny 2-2-1 network on XOR right here, step by step.

Section 01 · The problem

Why a network needs to learn

A neural network is just a stack of linear maps separated by simple nonlinearities like sigmoid or ReLU. With the right weights it can approximate any function — that is the universal approximation theorem. The catch is the words 'with the right weights'. For any nontrivial task, the right weights are sitting in a million-dimensional haystack and no amount of analytical work will find them. They have to be learned from examples: shown an input, shown the desired output, allowed to adjust. Backpropagation is the adjustment rule that makes this possible at scale.

Section 02 · The forward pass

What the network thinks the answer is

Feed the input vector x into the first layer. Each neuron computes a weighted sum of its inputs, adds a bias, and pushes the result through its activation function. The outputs of layer 1 become the inputs of layer 2; the outputs of layer 2 become the inputs of layer 3; and so on. By the time you reach the output layer, the network has produced its current guess ŷ. No learning yet — just a long chain of multiplications and additions. The forward pass is fast and embarrassingly parallel; on a GPU it is one matrix multiply per layer.

Section 03 · The loss

How wrong it was

Compare the network's guess ŷ to the truth y and collapse the comparison into a single number L — the loss. For regression, L is the squared error (ŷ − y)². For classification, L is the cross-entropy −Σy·log ŷ. The loss is a scalar function of every weight in the network: change any weight, and L changes. The whole goal of training is to find weights that make L small. We cannot solve that minimisation directly — the landscape has too many dimensions — but we can compute the slope of L with respect to each weight, and follow it downhill.

Section 04 · The chain rule, walked backwards

How a tiny nudge propagates

The loss depends on the output, the output depends on the last layer's weights and its inputs, and those inputs depend on the previous layer's weights, recursively all the way to the input. The chain rule says: multiply the local derivatives along the path. Backprop is the bookkeeping that does this efficiently — one pass from the output back to the input, reusing each layer's already-computed activations and gradients. The result, for every weight w in the network, is ∂L/∂w: how much the loss would change if you nudged w by an infinitesimal amount. Linnainmaa wrote this down in a Finnish master's thesis in 1970; Rumelhart, Hinton and Williams gave it the modern form and the name in 1986.

Mini-net · 2 inputs → 2 hidden → 1 output · learning XOR

0 ⊕ 0 = 0

0.596

0 ⊕ 1 = 1

0.606

1 ⊕ 0 = 1

0.597

1 ⊕ 1 = 0

0.605

Edge thickness encodes |w|, amber means positive, cyan negative. Click Step a few times to see weights drift; Train ×200 watches the loss collapse from ~0.25 toward zero as the four XOR predictions snap into 0/1. Reset re-randomises and you can race the same descent again.

Section 05 · The update

Gradient descent — one tiny step downhill

Once you have ∂L/∂w for every weight, the update is straightforward: w ← w − η ∂L/∂w. Here η is the learning rate, a small positive number — typically 10⁻³ to 10⁻¹. Each step nudges every weight in the direction that locally lowers the loss, by an amount proportional to that direction's steepness. Do this once per mini-batch of examples, run for thousands of epochs, and the network's loss falls from random-guess (around 0.7 in nats for classification) down to near-zero. The landscape it descends is famously non-convex — full of saddles, plateaus, narrow ravines — but in practice gradient descent (plus its momentum-based cousins Adam and RMSProp) finds usable minima with reliability that still puzzles theorists.

Rumelhart · Hinton · Williams · 1986

w ← w − η ∇_w L

The single update rule that, scaled four orders of magnitude, became every modern neural network. Linnainmaa wrote the bookkeeping down first, in a 1970 Finnish master's thesis.

Section 06 · From this to GPT

Scaling the same algorithm

The 1986 paper trained nets with a few hundred weights to recognise simple shapes. GPT-4 has on the order of a trillion. Nothing about the chain rule changed. What changed is the substrate: GPUs and TPUs make each forward and backward pass a stack of tensor contractions that can run at hundreds of teraflops; mixed-precision arithmetic, gradient checkpointing, ZeRO sharding, and a dozen other tricks let you fit huge models into finite memory; and the data, scraped from the open internet, finally became plentiful enough to justify the parameter count. Same algorithm, four orders of magnitude more compute. That is the whole story of the deep-learning revolution — backprop, just scaled.

See it train

Open the Explorer.

The Explorer lets you choose a network topology, pick a toy task (XOR, two moons, circle-vs-square), drag the learning rate, and watch the decision boundary form in real time. The same algorithm that powers every transformer, training inside your browser tab.

→ Open the Explorer

Where you meet it

How and where this technique lives in the world today.

Deep learning

Every modern neural network — image classifier, language model, recommender — is trained by backpropagation. PyTorch and JAX implement it as their differentiation core (autograd).

Computer vision

Convolutional networks for medical imaging, autonomous driving and biometric ID all use backprop to learn filter kernels directly from labelled data; gradients flow back through convolutions and pooling layers.

Language models

GPT, Claude, Llama and every transformer are trained by backpropagating cross-entropy loss through trillions of parameters. The single algorithm that scales from one neuron to one trillion.

Robotics & control

Policy-gradient methods in reinforcement learning use backprop to update neural controllers from reward signals; modern bipedal-walking and dexterous-manipulation robots all use it.

Inverse problems in science

Physicists invert experiments by setting up the forward model in PyTorch/JAX and backpropagating through it — used in protein design (AlphaFold), differentiable rendering, and gradient-based experimental design.