Neural Networks

Running the forward pass…

DSAverse

Initializing Sorting Algorithms...

Sorting

Trees

Graphs

Preparing interactive visualizations...

Back to Neural Networks

Normalization

Subtract the mean, divide by the spread, then let γ and β put back what training wants. Batch norm and layer norm, computed on a batch small enough to read, and what they do to a deep network.

Builds onVanishing and Exploding Gradients Forward Pass

Every number computed in your browser from seeded random batches and weights

Act 1 of 5 · Batch normstep 1 of 17

X, the layer's input[6, 4]

	u1	u2	u3	u4
ex 1	0.8	−4.6	−0.6	0.6
ex 2	1.5	−4.3	−2.6	0.8
ex 3	2.4	−0.3	0.7	1.0
ex 4	2.0	0.7	−0.4	0.0
ex 5	3.6	−2.4	−2.4	1.0
ex 6	2.6	1.0	−0.1	−0.7

Speed

A layer's pre-activations for a batch of 6 examples: one row per example, one column per unit, so X has shape [6, 4]. Each unit lives on its own scale: unit 2's values span 5.60, unit 4's only 1.70, and their averages run from −1.65 to 2.15. Whatever comes next has to cope with all of that, and with it shifting as earlier layers train. Batch norm re-centres and rescales each unit using this batch's own numbers.

X \in \mathbb{R}^{6 \times 4}: \text{ one row per example, one column per unit}

The recipe

The same two lines; only the axis of μ and σ² changes.

\hat{x} = \frac{x - \mu}{\sqrt{\sigma^2 + \varepsilon}}

y = \gamma\,\hat{x} + \beta

	batch norm	layer norm
μ, σ² over	the batch, per unit	the units, per example
depends on batch mates	yes	no
at inference	running averages	same as training
used in	CNNs	transformers

A common misconception

“Batch norm normalises each example on its own.”

Its statistics come from the whole batch, so changing one example moves everyone else's output. Over 1,000 seeded batches, redrawing the last example moved example 1's x̂ by a median of:

batch size	batch norm	layer norm
2	0.5	0
4	0.34	0
8	0.15	0
32	0.036	0
128	0.0089	0

Bigger batches dilute the effect, which is one reason batch norm wants them; layer norm never has it.

Check yourself

Question 1 of 3

A batch of 32 examples passes through a layer of 64 units. Over which numbers does batch norm compute each mean and variance?

Normalization

Subtract the mean, divide by the spread, then let γ and β put back what training wants. Batch norm and layer norm, computed on a batch small enough to read, and what they do to a deep network.

Builds onVanishing and Exploding Gradients Forward Pass

Every number computed in your browser from seeded random batches and weights

Act 1 of 5 · Batch normstep 1 of 17

X, the layer's input[6, 4]

	u1	u2	u3	u4
ex 1	0.8	−4.6	−0.6	0.6
ex 2	1.5	−4.3	−2.6	0.8
ex 3	2.4	−0.3	0.7	1.0
ex 4	2.0	0.7	−0.4	0.0
ex 5	3.6	−2.4	−2.4	1.0
ex 6	2.6	1.0	−0.1	−0.7

Speed

X \in \mathbb{R}^{6 \times 4}: \text{ one row per example, one column per unit}

The recipe

The same two lines; only the axis of μ and σ² changes.

\hat{x} = \frac{x - \mu}{\sqrt{\sigma^2 + \varepsilon}}

y = \gamma\,\hat{x} + \beta

	batch norm	layer norm
μ, σ² over	the batch, per unit	the units, per example
depends on batch mates	yes	no
at inference	running averages	same as training
used in	CNNs	transformers

A common misconception

“Batch norm normalises each example on its own.”

Its statistics come from the whole batch, so changing one example moves everyone else's output. Over 1,000 seeded batches, redrawing the last example moved example 1's x̂ by a median of:

batch size	batch norm	layer norm
2	0.5	0
4	0.34	0
8	0.15	0
32	0.036	0
128	0.0089	0

Bigger batches dilute the effect, which is one reason batch norm wants them; layer norm never has it.

Check yourself

Question 1 of 3

A batch of 32 examples passes through a layer of 64 units. Over which numbers does batch norm compute each mean and variance?

DSAverse

Normalization

See also

Normalization

See also