Scaled Dot-Product Attention

softmax(QKᵀ / √dₖ) V, one matrix step at a time, with the shape at every step and the tiny GPT's real numbers in every cell. Then the √dₖ: what it is for, and what happens without it.

Builds onSelf-Attention Matrix Multiplication

Trained for this site · tiny GPT · 195,136 numbers

Loading the tiny GPT (390 KB)…

Shapes, for T tokens

Q, K, V [T, dₖ]
QKᵀ [T, dₖ] × [dₖ, T] = [T, T]
÷√dₖ, mask, softmax [T, T]
× V [T, T] × [T, dₖ] = [T, dₖ]

A common misconception

“The √d is a minor numerical detail.”

Without it, random 64-dimensional queries and keys give scores spread about 8 wide, and softmax hands the top key most of the weight while passing back a fraction of the gradient — see the Why √d step. In the trained tiny GPT, leaving it out moves the share of near one-hot rows from 8% to 45% and the loss from 0.328 to 0.399, measured over 8,701 predictions.

Check yourself

Question 1 of 3

Why are the scores divided by √dₖ?

DSAverse

Scaled Dot-Product Attention

See also

Scaled Dot-Product Attention

See also