Multi-Head Attention

Split 64 numbers into four heads of 16, let each attend on its own, join them and mix with W_O. The tiny GPT's real heads side by side — and what it loses when you switch one off.

Builds onCausal Masking

Trained for this site · tiny GPT · 195,136 numbers

Loading the tiny GPT (390 KB)…

Four steps

Project: Q, K, V = X W, each [T, 64].
Split: four column blocks of 16.
Attend: softmax(QₕKₕᵀ / √16 + M) Vₕ, per head.
Join and mix: concat to [T, 64], then × W_O.

A common misconception

“Four heads means four times the parameters.”

The tiny GPT's W_Q is one [64, 64] matrix, and so are W_K, W_V and W_O: 16,640 numbers per layer, exactly what a single 64-wide head would use. The heads are slices, not copies — and they are far from equal: switching off L1H1 takes the loss from 0.328 to 0.690, switching off L3H2 leaves it at 0.328.

Check yourself

Question 1 of 3

How do the four heads of a 64-wide layer get their queries?

DSAverse

Multi-Head Attention

See also

Multi-Head Attention

See also