DSAverse

Initializing Sorting Algorithms...

Sorting

Trees

Graphs

Preparing interactive visualizations...

Cheatsheet Review Compare Contact

Transformers

Paying attention…

DSAverse

Making data structures, algorithms, core computer science, mathematics and AI accessible through interactive visualizations.

Practice

Review
Compare
Complexity Cheatsheet
Home
Contact

Data Structures & Algorithms

Basics
Sorting
Searching
Trees
Graph Algorithms
Dynamic Programming
Indexing

CS Core

Operating Systems
Computer Networks
Database Systems
Under the Hood

Math

Combinatorics
Probability and Statistics
Vectors
Linear Algebra
Differential Calculus
Integral Calculus

AI & ML

Machine Learning
Neural Networks
Deep Learning Architectures
Transformers

© 2025 DSAverse. Built withfor better learning.

Back to Transformers

The Transformer Block

LayerNorm, attention, add; LayerNorm, feed-forward, add. Follow one token's row through a real block of the tiny GPT, watch the residual stream grow, and see what the model loses without the additions.

Builds onPositional Encoding

Trained for this site · tiny GPT · 195,136 numbers

Loading the tiny GPT (390 KB)…

Sizes in the tiny GPT

T = tokens in the input (up to 64)
d = 64 numbers per token
4 heads of 16
FFN: 64 → 256 → 64
3 blocks · 49,984 numbers each

A common misconception

“Each layer transforms the representation into a new one.”

Nothing in a block overwrites the stream; attention and the FFN each add a vector to it. The proof is what happens when the additions are replaced by assignments: the same trained weights go from loss 0.329 to 8.668 over 100 stories, and the top guess is right 1% of the time instead of 86%.

Check yourself

Question 1 of 3

In a pre-norm block like GPT-2's, what does the attention sublayer compute?

See also

More in this section

Inside a Tiny GPT Multi-Head Attention Scaled Dot-Product Attention Causal Masking