Byte-Pair Encoding

AI and Machine Learning · Language Models · 2015 · also: BPE, Subword tokenization · byte-pair-encoding.yaml

Merges frequent character pairs into subword units until the vocabulary is full. Any string stays representable, so rare and unseen words degrade into pieces instead of a single unknown token.

• supersedescorrects · extends
• classifiesspecializes · part-of
• substitutes forapproximates · alternative-to
• depends onrequires · validates

Colour is the family; a dashed line is the second member of it.

This node

• correctsfixes a defect in Word Embeddinga fixed word vocabulary cannot represent anything unseen

References

• Neural Machine Translation of Rare Words with Subword Units — Sennrich, Haddow, Birch — 2015 · link
• Background — Byte-pair encoding on Wikipedia