AI and Machine Learning

Language Models

Pretraining, scaling and alignment of models over text.

Colour is the family; a dashed line is the second member of it.

Drag to pan · scroll to zoom · click a node to open it · hover one for the reasons on its edges, and to bring it forward
G autoregressive-language-model Autoregressive Language Model bert BERT autoregressive-language-model->bert transformer Transformer autoregressive-language-model->transformer bert->transformer byte-pair-encoding Byte-Pair Encoding word-embedding Word Embedding byte-pair-encoding->word-embedding chinchilla-scaling Chinchilla Scaling scaling-laws Scaling Laws chinchilla-scaling->scaling-laws direct-preference-optimization Direct Preference Optimization rlhf RLHF direct-preference-optimization->rlhf reward-model Reward Model reward-model->rlhf rlhf->autoregressive-language-model supervised-fine-tuning Supervised Fine- Tuning rlhf->supervised-fine-tuning ppo Proximal Policy Optimization rlhf->ppo scaling-laws->autoregressive-language-model supervised-fine-tuning->autoregressive-language-model behavior-cloning Behavior Cloning supervised-fine-tuning->behavior-cloning verifiable-rewards Reinforcement Learning from Verifiable Rewards verifiable-rewards->rlhf grpo Group Relative Policy Optimization verifiable-rewards->grpo
The 15 reasons on these edges, as text

11 nodes

Word Embedding

Maps words to dense vectors trained so that context predicts the word. Gave the field the observation that direction in the space carries meaning. · 2013

Byte-Pair Encoding

Merges frequent character pairs into subword units until the vocabulary is full. Any string stays representable, so rare and unseen words degrade int… · 2015

BERT

An encoder-only transformer pretrained by masking tokens and predicting them from both sides. Showed that one pretrained model plus a small head beat… · 2018

Scaling Laws

Loss falls as a power law in parameters, data and compute, over many orders of magnitude. Turned model design into a budgeting exercise with a predic… · 2020

Supervised Fine-Tuning

Continues next-token training on curated demonstrations of the behaviour wanted. Cheap, stable, and bounded by how well the desired behaviour can be… · 2021

Chinchilla Scaling

Refits the compute-optimal frontier and finds parameters and tokens should grow together in roughly equal proportion. Implied the models of the day w… · 2022

Reward Model

A model trained on pairwise human preferences to score responses, standing in for the objective nobody can write down. Also the component that gets g… · 2022

RLHF

Fits a reward model to human preference comparisons, then optimises the language model against it. The step that turned a next-token predictor into s… · 2022

Direct Preference Optimization

Shows the preference objective can be optimised as a classification loss on the model itself, with no reward model and no rollouts. · 2023

Reinforcement Learning from Verifiable Rewards

Replaces the learned reward with a programmatic check — does the proof verify, do the tests pass, is the answer right. Works only where correctness i… · 2025

Autoregressive Language Model

A decoder-only transformer trained to predict the next token. The objective is trivial and the capability that falls out of scaling it was not antici…