Residual Connections

Twenty layers should beat four, yet without skips they cannot even fit the training data. Watch the points merge and the gradient vanish in a deep plain network, then add hₗ = hₗ₋₁ + f(hₗ₋₁) and train it again.

Builds onVanishing and Exploding Gradients Backpropagation

Trained in your browser · 8 units per layer · 1,473 numbers at 20 blocks

Training three networks in your browser (2,400 steps)… 0%

What the skip changes

Forward: a block adds a correction to h instead of replacing it, so it cannot erase the differences between inputs.
Backward: the derivative of hₗ₋₁ + f(hₗ₋₁) is I + ∂f/∂h, so the gradient from above reaches every layer.
Cost: one addition per unit. No new weights.
Every transformer block wraps both its attention and its feed-forward layer in one.

A common misconception

“Extra layers can only help: at worst they learn to pass their input through.”

Measuring once the networks have trained…

Check yourself

Question 1 of 3

A 20-block network without skips ends with training loss 0.693 on a two-class task. What does that tell you?

DSAverse

Residual Connections

See also

Residual Connections

See also