DSAverse

Initializing Sorting Algorithms...

Sorting

Trees

Graphs

Preparing interactive visualizations...

Back to Transformers

Query, Key and Value

Attention is a dictionary lookup made soft. Score a query against every key, turn the scores into weights, and blend the values. Four keys on a plane are enough to see all of it.

Builds onWhy Attention Dot Product

Hand-set keys and values, two numbers each

Act 1 of 7 · Exact Lookupstep 1 of 24

table = {

"k₁": v₁,

"k₂": v₂,

"k₃": v₃,matched

"k₄": v₄,

}

table["k₃"] → v₃

table["k₅"] → KeyError

Hard lookup: one exact match, one value, or nothing.

Soft lookup: every key matches a little; the answer is a blend of every value.

Speed

A Python dictionary is a lookup: give it a key and it returns that key's value. The query has to equal a key exactly — "k₃" returns v₃, and anything else is a KeyError. Attention is the same lookup made soft: every key gets a score for how well it matches the query, and the answer is a blend of all the values, mixed by those scores.

Three roles

query: what this position is looking for.
key: what each position offers to be matched on.
value: what each position hands over when it is chosen.
out = Σ softmax(q · kⱼ) vⱼ

A common misconception

“Attention picks the key closest to the query.”

It scores every key with a dot product, which counts length as well as direction, and then blends. Turn a length-2 query all the way round these four keys: for 11.9% of directions the best-scoring key is not the nearest in direction, and the top key takes 83.6% of the weight on average — the rest still flows in from the others.

Check yourself

Question 1 of 3

What does the score q · k reward?

DSAverse

Query, Key and Value

See also

Query, Key and Value

See also