Positional Encoding

Attention alone cannot tell "Mia saw Ben" from "Ben saw Mia". Learned position vectors, sinusoids whose dot product depends only on distance, and RoPE's rotations, checked on a real tiny GPT.

Builds onMulti-Head Attention

Trained for this site · tiny GPT · 195,136 numbers

Loading the tiny GPT (390 KB)…

Three ways to add order

Learned: one trained vector per position, added to the token (GPT-2, this tiny GPT). Stops at the table's length.
Sinusoidal: sin and cos of p·ωᵢ, added to the token (the 2017 Transformer). Nothing to train.
Rotary (RoPE): rotate q and k by their positions (LLaMA and most recent models). Scores see only the offset.

A common misconception

“The model reads tokens left to right, so it knows the order anyway.”

Every position is processed at once, and q · k compares content only. With position vectors and the causal mask removed, the tiny GPT gives a sentence and its reversal exactly the same prediction. Even with the mask on, removing positions takes its loss from 0.326 to 2.196 over 100 stories.

Check yourself

Question 1 of 3

Why does a transformer need position information at all?

DSAverse

Positional Encoding

See also

Positional Encoding

See also