Inside a Tiny GPT

A whole forward pass of a real 195,136-number GPT, from bytes to the next token: every table, every block, the scores for all 640 tokens. Then open it up yourself and inspect any head or any of its 768 neurons.

Builds onThe Transformer Block

Trained for this site · tiny GPT · 195,136 numbers

Loading the tiny GPT (390 KB)…

The tiny GPT

vocabulary 640 (byte-level BPE)
context 64 tokens
d = 64, 4 heads of 16
3 blocks, FFN 256
trained on 12,000 stories

A common misconception

“A language model looks up answers in what it memorised.”

There is no text stored anywhere in it: 195,136 numbers, and every prediction is the same sequence of lookups, matrix products, softmaxes and additions shown here. Switch off one layer's attention and those same numbers predict far worse: with layer 1's heads off, the top guess is right 54% of the time instead of 86%.

Check yourself

Question 1 of 3

How does the tiny GPT turn its final vectors into scores for the 640 tokens?

DSAverse

Inside a Tiny GPT

See also

Inside a Tiny GPT

See also