DSAverse

Initializing Sorting Algorithms...

Sorting

Trees

Graphs

Preparing interactive visualizations...

Back to Neural Networks

Neural Network Playground

Build a network, train it in your browser and look inside: every unit draws its own picture of the plane. The tour reads a trained network's first layer as straight edges and its second as their combinations, then starts every weight at the same value to show why networks start from random ones.

Builds onBackpropagation Optimizers Dropout and Weight Decay

Every network here is trained in your browser, from seeded data and weights

Training three networks in your browser… 0 of 3 done

The tour's network

h^{(l)} = f\big(W_l\,h^{(l-1)} + b_l\big)

layer	W	b	params
1 (tanh)	[8, 2]	[8]	24
2 (tanh)	[8, 8]	[8]	72
output (σ)	[1, 8]	[1]	9
total			105

Trained with Adam, η = 0.03, on all 200 points for 300 steps.

A common misconception

“Starting every weight at the same value is a neutral start; training will tell the units apart.”

It never does. Units that start equal get equal gradients forever. The tour's spiral over 100 seeds, 300 steps each:

start	train acc	units
random	100%	all different
all 0.5	57%	copies in 100/100
all 0	50%	frozen in 100/100

Check yourself

Question 1 of 3

A unit in the first hidden layer computes h = tanh(a·x + b·y + c). What does its picture over the plane look like?

Neural Network Playground

Builds onBackpropagation Optimizers Dropout and Weight Decay

Every network here is trained in your browser, from seeded data and weights

Training three networks in your browser… 0 of 3 done

The tour's network

h^{(l)} = f\big(W_l\,h^{(l-1)} + b_l\big)

layer	W	b	params
1 (tanh)	[8, 2]	[8]	24
2 (tanh)	[8, 8]	[8]	72
output (σ)	[1, 8]	[1]	9
total			105

Trained with Adam, η = 0.03, on all 200 points for 300 steps.

A common misconception

“Starting every weight at the same value is a neutral start; training will tell the units apart.”

It never does. Units that start equal get equal gradients forever. The tour's spiral over 100 seeds, 300 steps each:

start	train acc	units
random	100%	all different
all 0.5	57%	copies in 100/100
all 0	50%	frozen in 100/100

Check yourself

Question 1 of 3

A unit in the first hidden layer computes h = tanh(a·x + b·y + c). What does its picture over the plane look like?

DSAverse

Neural Network Playground

See also

Neural Network Playground

See also