DSAverse

Initializing Sorting Algorithms...

Sorting

Trees

Graphs

Preparing interactive visualizations...

Back to Machine Learning

Linear Regression

Two routes to the best line: roll a ball down the loss bowl with gradient descent, or solve the normal equations in one step. They arrive at the same place.

Builds onLeast Squares Gradient Descent Inverse Matrices

Gradient descent runs in your browser · seeded example points

Act 1 of 6 · The modelstep 1 of 21

Grab a point to take over in Explore.

Speed

Ten points and one line, ŷ = w·x + b, scored by the mean squared error. This line, w = 0.0 and b = −1.0, has loss 4.076. Linear regression asks for the w and b with the smallest loss — and there are two very different ways to find them.

L(w, b) = \frac{1}{10}\sum_{i=1}^{10} (w x_i + b - y_i)^2 = L(0.0,\, -1.0) = 4.076

Two routes, one line

Gradient descent — step downhill, θ ← θ − η∇L, until the slope is zero. Works for any differentiable model.
Normal equations — set the slope to zero on paper: XᵀXθ = Xᵀy. Exact, but only for models linear in θ, and only while XᵀX is small enough to solve.

X[n, 2] =[10, 2]XᵀX[2, 2] =[2, 2]θ[2, 1] =[2, 1]

A common misconception

“Gradient descent might get stuck somewhere else, so its answer is only approximate.”

Not here. The loss is a bowl with one bottom. This page ran gradient descent from 200 random starts across the whole (w, b) plane on the tour's points: every run finished within 10⁻⁹ of the normal-equations answer. Local minima belong to curvier losses — the neural networks later on.

Check yourself

Question 1 of 3

Why does gradient descent on linear regression’s mean squared error always reach the same line, wherever it starts?

Linear Regression

Two routes to the best line: roll a ball down the loss bowl with gradient descent, or solve the normal equations in one step. They arrive at the same place.

Builds onLeast Squares Gradient Descent Inverse Matrices

Gradient descent runs in your browser · seeded example points

Act 1 of 6 · The modelstep 1 of 21

Grab a point to take over in Explore.

Speed

L(w, b) = \frac{1}{10}\sum_{i=1}^{10} (w x_i + b - y_i)^2 = L(0.0,\, -1.0) = 4.076

Two routes, one line

Gradient descent — step downhill, θ ← θ − η∇L, until the slope is zero. Works for any differentiable model.
Normal equations — set the slope to zero on paper: XᵀXθ = Xᵀy. Exact, but only for models linear in θ, and only while XᵀX is small enough to solve.

X[n, 2] =[10, 2]XᵀX[2, 2] =[2, 2]θ[2, 1] =[2, 1]

A common misconception

“Gradient descent might get stuck somewhere else, so its answer is only approximate.”

Check yourself

Question 1 of 3

Why does gradient descent on linear regression’s mean squared error always reach the same line, wherever it starts?

DSAverse

Linear Regression

See also

Linear Regression

See also