DSAverse

Initializing Sorting Algorithms...

Sorting

Trees

Graphs

Preparing interactive visualizations...

Back to Neural Networks

Forward Pass

Follow two numbers through a trained 2-3-2 network that solves XOR: three weighted sums, three ReLUs, two scores, one softmax. Then the same arithmetic for a whole batch, and for every point of the plane at once.

Builds onActivation Functions Matrix Multiplication

Trained for this page, rounded to one decimal

Act 1 of 6 · The networkstep 1 of 25

Speed

This network has 2 inputs, 3 hidden ReLU units and 2 outputs, and it was trained on the XOR data the perceptron could not separate. It classifies 116 of the 120 training points correctly, and 927 of 1,000 fresh points drawn the same way (92.7%).

p = \operatorname{softmax}\big(W_2\,\operatorname{ReLU}(W_1 x + b_1) + b_2\big)

The pieces

piece	shape	does
x	[2]	the input
W₁	[3, 2]	one row per hidden unit
b₁	[3]	shifts each unit
ReLU	keeps shape	zeroes negatives
W₂	[2, 3]	one row per output
b₂	[2]	shifts each score
softmax	keeps shape	scores → probabilities

A common misconception

“A softmax output of 0.99 means the network is 99% sure, so it must know.”

This network only ever saw points within about 1.2 of the centre. On a circle of radius 5, where there is no data at all, it still gives one class at least 0.99 in 78.6% of 360 directions (mean top probability 0.985). With ReLU, the scores grow in a straight line as you move away, so softmax saturates. Of 100 retrainings from other seeds, the 26 that reached 90% accuracy had a median share of 81%. Softmax measures how far a point is from the boundary, not how much the network knows. Switch Explore to “5× wider” to see it.

Check yourself

Question 1 of 3

A layer takes 3 inputs and has 4 units. In h = f(Wx + b), what shape is W?

Forward Pass

Builds onActivation Functions Matrix Multiplication

Trained for this page, rounded to one decimal

Act 1 of 6 · The networkstep 1 of 25

Speed

p = \operatorname{softmax}\big(W_2\,\operatorname{ReLU}(W_1 x + b_1) + b_2\big)

The pieces

piece	shape	does
x	[2]	the input
W₁	[3, 2]	one row per hidden unit
b₁	[3]	shifts each unit
ReLU	keeps shape	zeroes negatives
W₂	[2, 3]	one row per output
b₂	[2]	shifts each score
softmax	keeps shape	scores → probabilities

A common misconception

“A softmax output of 0.99 means the network is 99% sure, so it must know.”

Check yourself

Question 1 of 3

A layer takes 3 inputs and has 4 units. In h = f(Wx + b), what shape is W?

DSAverse

Forward Pass

See also

Forward Pass

See also