DSAverse

Initializing Sorting Algorithms...

Sorting

Trees

Graphs

Preparing interactive visualizations...

Back to Neural Networks

Dropout and Weight Decay

Train a network with far more parameters than data and it memorises the noise. Two cures, trained live: shrink every weight a little on every step, or switch off a random half of the units on every step.

Builds onOverfitting and Underfitting Optimizers

Every network here is trained in your browser, from seeded data and weights

Training three networks in your browser… 0 of 3 done

The two rules

Weight decay, every step

w \leftarrow (1 - \eta\lambda)\,w - \eta\,\partial L_{\text{data}}/\partial w

Dropout, training only

\tilde h = m \odot h / (1 - p), \quad m_k \sim \text{Bernoulli}(1 - p)

here	λ = 0.01	p = 0.5
in PyTorch	weight_decay=λ	nn.Dropout(p)
at test	nothing to do	model.eval()

A common misconception

“Regularisation helps because the network fits its training data better.”

It fits it worse, on purpose. Medians over 100 seeds of this page's experiment, after 1,000 steps:

	train loss	val loss	val acc
plain	0.007	1.440	78%
λ = 0.01	0.297	0.451	81%
p = 0.5	0.221	0.565	81%

Lower validation loss than the plain run in 100 of 100 seeds with weight decay and 97 of 100 with dropout.

Check yourself

Question 1 of 3

A network was trained with dropout p = 0.5, the inverted dropout PyTorch uses. At test time, what happens to a hidden unit's activation h?

Dropout and Weight Decay

Builds onOverfitting and Underfitting Optimizers

Every network here is trained in your browser, from seeded data and weights

Training three networks in your browser… 0 of 3 done

The two rules

Weight decay, every step

w \leftarrow (1 - \eta\lambda)\,w - \eta\,\partial L_{\text{data}}/\partial w

Dropout, training only

\tilde h = m \odot h / (1 - p), \quad m_k \sim \text{Bernoulli}(1 - p)

here	λ = 0.01	p = 0.5
in PyTorch	weight_decay=λ	nn.Dropout(p)
at test	nothing to do	model.eval()

A common misconception

“Regularisation helps because the network fits its training data better.”

It fits it worse, on purpose. Medians over 100 seeds of this page's experiment, after 1,000 steps:

	train loss	val loss	val acc
plain	0.007	1.440	78%
λ = 0.01	0.297	0.451	81%
p = 0.5	0.221	0.565	81%

Lower validation loss than the plain run in 100 of 100 seeds with weight decay and 97 of 100 with dropout.

Check yourself

Question 1 of 3

A network was trained with dropout p = 0.5, the inverted dropout PyTorch uses. At test time, what happens to a hidden unit's activation h?

DSAverse

Dropout and Weight Decay

See also

Dropout and Weight Decay

See also