Autoregressive Language Model

AI and Machine Learning · Language Models · also: GPT, Decoder-only · autoregressive-language-model.yaml

A decoder-only transformer trained to predict the next token. The objective is trivial and the capability that falls out of scaling it was not anticipated.

• supersedescorrects · extends
• classifiesspecializes · part-of
• substitutes forapproximates · alternative-to
• depends onrequires · validates

Colour is the family; a dashed line is the second member of it.

This node

• specializesis a specific case of Transformerdecoder only, predicting forward with a causal mask
• alternative-tois a competing approach to BERTgenerates continuations rather than encoding a whole span

Referenced by

• extendsRLHF adds capability to thisaligns to preferences that next-token loss cannot express
• validatesScaling Laws is a test for thispredicts the loss a given budget will reach
• extendsSupervised Fine-Tuning adds capability to thistrains on demonstrations of the behaviour wanted

References

• Background — Generative pre-trained transformer on Wikipedia