Positional Encoding

AI and Machine Learning · Sequence Models · 2017 · positional-encoding.yaml

Injects each token's position into its representation, since attention alone sees an unordered set. Sinusoidal at first, later usually learned or applied as a rotation.

• supersedescorrects · extends
• classifiesspecializes · part-of
• substitutes forapproximates · alternative-to
• depends onrequires · validates

Colour is the family; a dashed line is the second member of it.

This node

• part-ofis a subtopic of Transformer

References

• Attention Is All You Need — Vaswani et al. — 2017 · link