DSAverse

Initializing Sorting Algorithms...

Sorting

Trees

Graphs

Preparing interactive visualizations...

Back to Neural Networks

How Networks Bend Space

A trained network does not draw a curve around the data. Each layer stretches, shifts and squashes the whole plane, and after the last one a straight line is enough. Watch every point and every grid line move, then meet a tangle that two units per layer can never undo.

Builds onForward Pass Matrix as a Transformation

Trained for this page, rounded

Act 1 of 8 · Tangledstep 1 of 17

xW₁+ b₁tanhW₂+ b₂tanhcut

Speed

160 points in two interlocking moons, 80 of each colour. No straight line splits them: trying every direction and every offset, the best line (dashed) gets 144 of 160 right. The white ring is P = (0.9, −0.5), a point this tour follows.

\text{best line: } 0.21\,x_1 - 0.98\,x_2 = -0.09 \;\Rightarrow\; 144 / 160 \text{ right}

The pieces

piece	does to the plane
W (stretch)	rotates, scales and shears; lines stay straight
+ b (shift)	slides everything the same way
tanh (squash)	pulls each coordinate into (−1, 1); the only bend
v · h + c	one straight cut in the last space
units per layer	the dimensions there are to pull apart in

A common misconception

“Stack enough layers and a network can untangle anything.”

We trained 80 networks on the ring with hidden layers of 2 units: 1, 2, 3 or 4 layers, 20 seeds each. Not one puts every point of the true circles on the right side; the best gets 98.1%, and each depth’s median is 82–89%. 3 of them still score 100% on their 160 training points, by threading the line through a gap between samples. With a single layer of 3 units, 15 of 20 get every point. Depth adds bends; only width adds room.

Check yourself

Question 1 of 3

A layer computes tanh(Wx + b). Which of its three parts can turn a straight grid line into a curve?

How Networks Bend Space

Builds onForward Pass Matrix as a Transformation

Trained for this page, rounded

Act 1 of 8 · Tangledstep 1 of 17

xW₁+ b₁tanhW₂+ b₂tanhcut

Speed

\text{best line: } 0.21\,x_1 - 0.98\,x_2 = -0.09 \;\Rightarrow\; 144 / 160 \text{ right}

The pieces

piece	does to the plane
W (stretch)	rotates, scales and shears; lines stay straight
+ b (shift)	slides everything the same way
tanh (squash)	pulls each coordinate into (−1, 1); the only bend
v · h + c	one straight cut in the last space
units per layer	the dimensions there are to pull apart in

A common misconception

“Stack enough layers and a network can untangle anything.”

Check yourself

Question 1 of 3

A layer computes tanh(Wx + b). Which of its three parts can turn a straight grid line into a curve?

DSAverse

How Networks Bend Space

See also

How Networks Bend Space

See also