DSAverse

Initializing Sorting Algorithms...

Sorting

Trees

Graphs

Preparing interactive visualizations...

Back to Neural Networks

Computational Graphs

How autograd gets a gradient without ever writing a derivative down: each operation records its local derivatives as it runs, and the backward pass replays the records newest first, multiplying as it goes.

Builds onForward Pass Derivative Rules

Every number computed in your browser by the autograd engine

Act 1 of 5 · Recordstep 1 of 25

Top line: the value. ∂ blue / ∂ orange: a positive / negative gradient of the loss with respect to that box.Amber outline: a parameter training would change.

Speed

Four numbers: a weight w = 1.1, a bias b = 0.6, an input x = 0.5 and its target y = 1.7. Four lines of code compute L = (wx + b − y)², the squared error of a one-weight model. As each line runs, the autograd engine computes the value and also writes a record on a tape. The dashed boxes mark where those records will go; nothing is there until its line has run.

L = (wx + b - y)^2 = \big(1.1\cdot0.5 + 0.6 - 1.7\big)^2

The tape

written top to bottom

Empty: no line has run yet.

Every local rule this graph needs

Each record stores these numbers when its line runs.

operation	local derivatives
a + b	1 and 1
a − b	1 and −1
a × b	b and a
u²	2u

Replay: newest record first, gradient in += gradient out × local derivative.

A common misconception

“A value gets its gradient from the operation that uses it.”

Only when one operation uses it. With the penalty, w feeds two records and its gradient is the sum of both. Over 1,000 random settings of w, x, b and y between −2 and 2, the summed gradient matched a measured slope every time (largest gap 8e-10). Keeping only the path replayed last, as a backward pass that assigns instead of adding would, missed by a median of 29% and had the wrong sign 125 times. Explore's “Overwrite instead of add” shows it.

Check yourself

Question 1 of 3

As the forward pass runs, what does each operation record for the backward pass?

Computational Graphs

Builds onForward Pass Derivative Rules

Every number computed in your browser by the autograd engine

Act 1 of 5 · Recordstep 1 of 25

Top line: the value. ∂ blue / ∂ orange: a positive / negative gradient of the loss with respect to that box.Amber outline: a parameter training would change.

Speed

L = (wx + b - y)^2 = \big(1.1\cdot0.5 + 0.6 - 1.7\big)^2

The tape

written top to bottom

Empty: no line has run yet.

Every local rule this graph needs

Each record stores these numbers when its line runs.

operation	local derivatives
a + b	1 and 1
a − b	1 and −1
a × b	b and a
u²	2u

Replay: newest record first, gradient in += gradient out × local derivative.

A common misconception

“A value gets its gradient from the operation that uses it.”

Check yourself

Question 1 of 3

As the forward pass runs, what does each operation record for the backward pass?

DSAverse

Computational Graphs

The tape

See also

Computational Graphs

The tape

See also