Causal Masking

A GPT learns to predict every next token of a sentence in one pass. A triangle of −∞ is what keeps each position from reading its own answer. Edit a word and watch exactly which predictions can change.

Builds onScaled Dot-Product Attention

Trained for this site · tiny GPT · 195,136 numbers

Loading the tiny GPT (390 KB)…

The rule

Output p is computed from tokens 1 to p, never from p + 1 onward.
Training predicts all T next tokens in one pass; the mask keeps every one of them honest.
Generation adds one token at a time and gets the same numbers, so earlier keys and values can be cached.

A common misconception

“A GPT reads a sentence one token at a time, left to right.”

On a prompt, and in training, it reads all of them at once: one pass, every position in parallel. What makes it behave as if it read left to right is the mask. Run any prefix on its own and its last position matches the full pass exactly — and editing each of 100 stories' last token changes 0 of 5,859 earlier outputs.

Check yourself

Question 1 of 3

What does the causal mask do to the attention scores?

DSAverse

Causal Masking

See also

Causal Masking

See also