DSAverse

Initializing Sorting Algorithms...

Sorting

Trees

Graphs

Preparing interactive visualizations...

Back to Differential Calculus

Gradient Descent

The algorithm that trains neural networks: feel the slope, step downhill, repeat. The learning rate decides whether that walk crawls, converges, zigzags or blows up.

η = 0.3 · step 1 of 40

Loss surface: loss = 0.1x² + 0.4y²

The same walk on the contour map

Loss after each step

Speed

A loss surface: low is good. Training a model means walking downhill on it. Gradient descent looks at the slope where it stands, ∇loss, and takes a step the other way, scaled by the learning rate η. The ball starts at (−2.6, 1.5), loss 1.576.

\nabla \text{loss}(x, y) = (0.2x,\; 0.8y)

\begin{pmatrix} x \\ y \end{pmatrix} \leftarrow \begin{pmatrix} -2.6 \\ 1.5 \end{pmatrix} - 0.3\begin{pmatrix} -0.52 \\ 1.2 \end{pmatrix} = \begin{pmatrix} -2.444 \\ 1.14 \end{pmatrix}

x \times (1 - 0.2\eta) = x \times \textcolor{#4ade80}{0.94}, \qquad y \times (1 - 0.8\eta) = y \times \textcolor{#4ade80}{0.76}

Gradient Descent

The algorithm that trains neural networks: feel the slope, step downhill, repeat. The learning rate decides whether that walk crawls, converges, zigzags or blows up.

η = 0.3 · step 1 of 40

Loss surface: loss = 0.1x² + 0.4y²

The same walk on the contour map

Loss after each step

Speed

\nabla \text{loss}(x, y) = (0.2x,\; 0.8y)

\begin{pmatrix} x \\ y \end{pmatrix} \leftarrow \begin{pmatrix} -2.6 \\ 1.5 \end{pmatrix} - 0.3\begin{pmatrix} -0.52 \\ 1.2 \end{pmatrix} = \begin{pmatrix} -2.444 \\ 1.14 \end{pmatrix}

x \times (1 - 0.2\eta) = x \times \textcolor{#4ade80}{0.94}, \qquad y \times (1 - 0.8\eta) = y \times \textcolor{#4ade80}{0.76}

DSAverse

Gradient Descent

See also

Gradient Descent

See also