DSAverse

Initializing Sorting Algorithms...

Sorting

Trees

Graphs

Preparing interactive visualizations...

Back to Neural Networks

Vanishing and Exploding Gradients

Backprop multiplies the gradient by one factor per layer. Follow it back through twelve layers and watch the product shrink to nothing or blow up, then see the weight scale that keeps it alive.

Builds onBackpropagation Activation Functions

Every number computed in your browser from seeded random weights

Act 1 of 5 · One pathone unit per layer, tanhstep 1 of 16

Speed

Start with the thinnest network there is: twelve layers of one tanh unit each. The forward pass is done: x = h₀ = −0.36 goes in, each layer multiplies by its weight and squashes, and out comes s = h₁₂ = 0.7872. Backprop starts from ∂s/∂h₁₂ = 1 and walks back one link at a time.

\begin{gathered} h_k = \tanh(w_k\, h_{k-1}), \qquad s = h_{12} \\ \frac{\partial s}{\partial h_{12}} = 1 \end{gathered}

One layer of backprop

From the gradient at layer k to the one below it.

u_k = f'(z_k) \odot \delta_k

\delta_{k-1} = W_k^{\top} u_k

\mathbb{E}\,\|\delta_{k-1}\|^2 \approx n\,\mathrm{Var}(w)\,\mathbb{E}[f'(z)^2]\;\|\delta_k\|^2

activation	f′(z)	scale that keeps it
Sigmoid	≤ 0.25	none that is sensible
tanh	≤ 1	Xavier, Var 1/n
ReLU	0 or 1	He, Var 2/n

A common misconception

“Switching to ReLU fixes vanishing gradients.”

ReLU removes saturation; the weight scale still decides. Over 1,000 seeded 12-layer nets 8 units wide, ReLU at PyTorch's default scale left a median ‖δ₀‖ of 2.2 × 10⁻⁶ (sigmoid: 2.6 × 10⁻¹¹). The same nets at He's scale: 0.1, exactly 6⁶ = 46,656 times larger, because without biases a ReLU net's gradient scales with the 12th power of the weights' scale. In 35 of the nets one layer had every unit off and the gradient was exactly 0 at every scale.

Check yourself

Question 1 of 3

Backprop multiplies the gradient by one factor per layer. If each of 12 factors is about 0.5, roughly what is left of a gradient that started at 1?

Vanishing and Exploding Gradients

Backprop multiplies the gradient by one factor per layer. Follow it back through twelve layers and watch the product shrink to nothing or blow up, then see the weight scale that keeps it alive.

Builds onBackpropagation Activation Functions

Every number computed in your browser from seeded random weights

Act 1 of 5 · One pathone unit per layer, tanhstep 1 of 16

Speed

\begin{gathered} h_k = \tanh(w_k\, h_{k-1}), \qquad s = h_{12} \\ \frac{\partial s}{\partial h_{12}} = 1 \end{gathered}

One layer of backprop

From the gradient at layer k to the one below it.

u_k = f'(z_k) \odot \delta_k

\delta_{k-1} = W_k^{\top} u_k

\mathbb{E}\,\|\delta_{k-1}\|^2 \approx n\,\mathrm{Var}(w)\,\mathbb{E}[f'(z)^2]\;\|\delta_k\|^2

activation	f′(z)	scale that keeps it
Sigmoid	≤ 0.25	none that is sensible
tanh	≤ 1	Xavier, Var 1/n
ReLU	0 or 1	He, Var 2/n

A common misconception

“Switching to ReLU fixes vanishing gradients.”

Check yourself

Question 1 of 3

Backprop multiplies the gradient by one factor per layer. If each of 12 factors is about 0.5, roughly what is left of a gradient that started at 1?

DSAverse

Vanishing and Exploding Gradients

See also

Vanishing and Exploding Gradients

See also