Neural Networks

Running the forward pass…

DSAverse

Initializing Sorting Algorithms...

Sorting

Trees

Graphs

Preparing interactive visualizations...

Back to Neural Networks

Optimizers

SGD, momentum, RMSProp and Adam start from the same point on the same narrow ravine. Step through each update rule, then race them, then turn the ravine and see which ones notice.

Builds onGradient Descent Backpropagation

Every step computed in your browser with torch.optim's update rules

Act 1 of 5 · SGD

step 1 of 29

Grey rings: equal loss. Dashed yellow ring: loss 0.001, the floor. White dot: the start.

Speed

A loss shaped like a ravine: f(x, y) = 0.01x² + 2y², steep across y (curvature 4) and almost flat along x (curvature 0.02, 200 times less). The minimum is at (0, 0). All four optimizers start at (−2.6, 1.1), loss 2.4876, where the gradient is (−0.052, 4.4): 85 times more of it points across the ravine than along it.

\begin{gathered} f(x, y) = 0.01x^2 + 2y^2 \\ \nabla f = (0.02x,\ 4y) = \left(-0.052,\ 4.4\right) \end{gathered}

The four update rules

At the tour's settings; ε = 10⁻⁸ keeps the division safe.

SGD

\theta \leftarrow \theta - 0.45\, \nabla f

Momentum

m \leftarrow 0.9\, m + \nabla f

\theta \leftarrow \theta - 0.15\, m

RMSProp

v \leftarrow 0.99\, v + 0.01\, (\nabla f)^2

\theta \leftarrow \theta - 0.03\, \nabla f / (\sqrt{v} + \varepsilon)

Adam

m \leftarrow 0.9\, m + 0.1\, \nabla f

v \leftarrow 0.999\, v + 0.001\, (\nabla f)^2

\theta \leftarrow \theta - 0.1\, \hat m / (\sqrt{\hat v} + \varepsilon)

A common misconception

“Adam adapts to the shape of the loss surface.”

It adapts to the axes. Over 1,000 starts, the ravine turned 45° (start turned with it) left SGD at a median 222 steps and momentum at 70, exactly as before for every start, while Adam went from 61 to 330 and RMSProp from 31 to 363 (median steps to reach the floor). Retuning does not rescue it: over learning rates from 0.1 to 5, Adam's best there (η = 0.9) needs 76, still more than momentum.

Check yourself

Question 1 of 3

On the ravine f = 0.01x² + 2y², why can't plain SGD simply use a bigger learning rate to move faster along the floor?

Optimizers

SGD, momentum, RMSProp and Adam start from the same point on the same narrow ravine. Step through each update rule, then race them, then turn the ravine and see which ones notice.

Builds onGradient Descent Backpropagation

Every step computed in your browser with torch.optim's update rules

Act 1 of 5 · SGD

step 1 of 29

Grey rings: equal loss. Dashed yellow ring: loss 0.001, the floor. White dot: the start.

Speed

\begin{gathered} f(x, y) = 0.01x^2 + 2y^2 \\ \nabla f = (0.02x,\ 4y) = \left(-0.052,\ 4.4\right) \end{gathered}

The four update rules

At the tour's settings; ε = 10⁻⁸ keeps the division safe.

SGD

\theta \leftarrow \theta - 0.45\, \nabla f

Momentum

m \leftarrow 0.9\, m + \nabla f

\theta \leftarrow \theta - 0.15\, m

RMSProp

v \leftarrow 0.99\, v + 0.01\, (\nabla f)^2

\theta \leftarrow \theta - 0.03\, \nabla f / (\sqrt{v} + \varepsilon)

Adam

m \leftarrow 0.9\, m + 0.1\, \nabla f

v \leftarrow 0.999\, v + 0.001\, (\nabla f)^2

\theta \leftarrow \theta - 0.1\, \hat m / (\sqrt{\hat v} + \varepsilon)

A common misconception

“Adam adapts to the shape of the loss surface.”

Check yourself

Question 1 of 3

On the ravine f = 0.01x² + 2y², why can't plain SGD simply use a bigger learning rate to move faster along the floor?

DSAverse

Optimizers

See also

Optimizers

See also