Why Attention

A seq2seq decoder has to remember a whole sentence in one vector, and on long sentences it cannot. Give it every encoder state and a way to choose among them, and watch the same small translator read the sentence word by word as it writes.

Builds onSequence-to-Sequence

Trained for this site · GRU encoder–decoder with attention · 10,256 numbers

Loading the trained translators (172 KB)…

One attention step

Score: the decoder state against every encoder state, eₜⱼ = hₜ · hⱼ.
Weigh: softmax, so the weights are positive and add to 1.
Read: the weighted average of the encoder states, cₜ.
Write: the next word from [hₜ; cₜ]. A transformer does the same with queries, keys and values, and nothing recurrent.

A common misconception

“Attention gives the decoder a better summary of the sentence.”

Measuring once the models have loaded…

Check yourself

Question 1 of 3

With attention, what does the decoder get at each output step that a plain encoder–decoder does not?

DSAverse

Why Attention

See also

Why Attention

See also