Self-Attention

Each token rebuilds itself from the tokens it finds relevant. Watch one head of a real, trained model work out who “She” is — every number computed in your browser.

Builds onDot Product Matrix Multiplication

Tiny GPT · 195K parameters · trained on 12,000 simple stories

Loading the tiny GPT — 390 KB of weights, fetched once

One head, four moves

Project each token into a query, a key and a value.
Score every query against every visible key: q · k / √dₖ.
Softmax each row into weights that add up to 1.
Blend the values with those weights.

X[T, d] =[T, 64]Q K V[T, dₖ] =[T, 16]weights[T, T] =[T, T]

A common misconception

“Attention picks the one word that matters.”

It takes a weighted average of every visible token. “She” putting 98% on “Mia” is unusually sharp: across 300 stories, the largest weight a token gives any one other token averages between 0.28 and 0.54, depending on the head. Most of the time the output is a genuine blend.

Check yourself

Question 1 of 3

In self-attention, what is the score between token i and an earlier token j?

DSAverse

Self-Attention

See also

Self-Attention

See also