Attention

AI and Machine Learning · Sequence Models · 2014 · attention.yaml

Compute a weighted sum over all positions, with weights from a learned compatibility score. Every position can reach every other in one hop, regardless of distance.

• supersedescorrects · extends
• classifiesspecializes · part-of
• substitutes forapproximates · alternative-to
• depends onrequires · validates

Colour is the family; a dashed line is the second member of it.

This node

• correctsfixes a defect in Recurrent Networka fixed-size hidden state cannot carry a whole sequence

Referenced by

• correctsFlashAttention fixes a defect in thisit is memory-bandwidth bound, not compute bound
• requiresTransformer does not work without thisit is the only operation left that mixes positions

References

• Neural Machine Translation by Jointly Learning to Align and Translate — Bahdanau, Cho, Bengio — 2014 · link
• Background — Attention (machine learning) on Wikipedia