DSAverse

Initializing Sorting Algorithms...

Sorting

Trees

Graphs

Preparing interactive visualizations...

Back to Neural Networks

Backpropagation

How a network finds out which way to adjust every one of its weights: the chain rule, run backwards through the graph of operations, one edge at a time.

Builds onDerivative Rules Partial Derivatives and Gradients Gradient Descent

Every number computed in your browser

Act 1 of 5 · Forwardstep 1 of 28

Top line: the value. ∂ blue / ∂ orange: a positive / negative gradient ∂L/∂(that box).Amber outline: a parameter.

Speed

A network with one input and two hidden units. x = 1.5000 goes in, and we want ŷ to come out as y = 0.5000. The amber boxes are the six parameters training can change. Every box is one operation the autograd engine recorded as it ran — this is its real graph, not a drawing of one.

L = (\hat y - y)^2, \quad \hat y = v_1 \tanh(w_1x + b_1) + v_2 \tanh(w_2x + b_2)

The whole backward pass in five rules

Every edge: gradient in = upstream × local.
a + b — local derivative 1: passes the gradient through.
a × b — local derivative is the other input.
tanh(z) — local derivative 1 − tanh²(z): near ±1, almost nothing gets through.
Branches: a value used twice adds up the gradients from both uses.

A common misconception

“To find a weight's gradient, nudge that weight and rerun the network.”

That is finite differences — the Check act does exactly it, and it needs two forward passes for every parameter. Backprop reuses the values stored on the way forward and gets every gradient in one backward sweep. For a 7-billion-parameter model that is the difference between 14 billion forward passes and the work of about three.

Check yourself

Question 1 of 3

During backpropagation a node receives ∂L/∂out from the node above it. What does it send back to each of its inputs?

Backpropagation

How a network finds out which way to adjust every one of its weights: the chain rule, run backwards through the graph of operations, one edge at a time.

Builds onDerivative Rules Partial Derivatives and Gradients Gradient Descent

Every number computed in your browser

Act 1 of 5 · Forwardstep 1 of 28

Top line: the value. ∂ blue / ∂ orange: a positive / negative gradient ∂L/∂(that box).Amber outline: a parameter.

Speed

L = (\hat y - y)^2, \quad \hat y = v_1 \tanh(w_1x + b_1) + v_2 \tanh(w_2x + b_2)

The whole backward pass in five rules

Every edge: gradient in = upstream × local.
a + b — local derivative 1: passes the gradient through.
a × b — local derivative is the other input.
tanh(z) — local derivative 1 − tanh²(z): near ±1, almost nothing gets through.
Branches: a value used twice adds up the gradients from both uses.

A common misconception

“To find a weight's gradient, nudge that weight and rerun the network.”

Check yourself

Question 1 of 3

During backpropagation a node receives ∂L/∂out from the node above it. What does it send back to each of its inputs?

DSAverse

Backpropagation

See also

Backpropagation

See also