The seq2seq decoder had one vector to remember a whole sentence. Give it every encoder state and a way to choose, and the same toy translator goes from 44% to 100% of words right on long sentences.

Drag a query arrow among four keys and watch the dot products, the softmax and the blended value move with it. Attention is a soft dictionary lookup, and here it fits on one plane.

Run a sentence through a small GPT trained for this site and follow one real head: the scores, the softmax and the blend that let "She" find the name it refers to.

The whole computation as four matrix steps with their shapes: QKᵀ, ÷√d, mask, softmax, ×V. Then raise the head size and watch softmax saturate when the √d is left out.

Why a GPT may only look back. Remove the triangle of −∞ and every earlier prediction changes when you edit a later word — the model would be reading the answer it is meant to predict.

Split 64 numbers into four heads of 16, let each attend on its own, join them and mix with W_O. The tiny GPT's real heads, side by side, and what the model loses when you switch one off.

Attention ignores order until you add it. Sinusoids, the tiny GPT's learned positions, rotary embeddings, and what happens to its predictions when the positions are taken away.

Attention, add and norm, feed-forward, add and norm, with the shape [T, d] at every arrow. Watch the residual stream carry each token through the block and what attention and the FFN each add to it.

BERT reads both ways, GPT only looks back, T5 does both and joins them with cross-attention: the decoder's queries against the encoder's keys and values. Three mask patterns, one mechanism.

The capstone: a full forward pass through a real 195,136-parameter GPT, from token ids to the next-word probabilities. Open any head or any feed-forward neuron and see what it does.