AI and Machine Learning

Language Models

Pretraining, scaling and alignment of models over text.

Colour is the family; a dashed line is the second member of it.

Drag to pan · scroll to zoom · click a node to open it
G autoregressive-language-model Autoregressive Language Model bert BERT autoregressive-language-model->bert generates continuations rather than encoding a whole span transformer Transformer autoregressive-language-model->transformer decoder only, predicting forward with a causal mask bert->transformer encoder only, trained by masking rather than predicting byte-pair-encoding Byte-Pair Encoding word-embedding Word Embedding byte-pair-encoding->word-embedding a fixed word vocabulary cannot represent anything unseen chinchilla-scaling Chinchilla Scaling scaling-laws Scaling Laws chinchilla-scaling->scaling-laws the earlier fits left models too large for their data direct-preference-optimization Direct Preference Optimization rlhf RLHF direct-preference-optimization->rlhf a reward model and an RL loop to fit a preference dataset reward-model Reward Model reward-model->rlhf rlhf->autoregressive-language-model aligns to preferences that next-token loss cannot express supervised-fine-tuning Supervised Fine- Tuning rlhf->supervised-fine-tuning the policy is initialised from a supervised pass ppo Proximal Policy Optimization rlhf->ppo the reward model is not differentiable through text scaling-laws->autoregressive-language-model predicts the loss a given budget will reach supervised-fine-tuning->autoregressive-language-model trains on demonstrations of the behaviour wanted verifiable-rewards Reinforcement Learning from Verifiable Rewards verifiable-rewards->rlhf a learned reward can be gamed; a checked answer cannot grpo Group Relative Policy Optimization verifiable-rewards->grpo one scalar per completion, with no critic to fit

11 nodes

Word Embedding

Maps words to dense vectors trained so that context predicts the word. Gave the field the observation that direction in the space carries meaning. · 2013

Byte-Pair Encoding

Merges frequent character pairs into subword units until the vocabulary is full. Any string stays representable, so rare and unseen words degrade int… · 2015

BERT

An encoder-only transformer pretrained by masking tokens and predicting them from both sides. Showed that one pretrained model plus a small head beat… · 2018

Scaling Laws

Loss falls as a power law in parameters, data and compute, over many orders of magnitude. Turned model design into a budgeting exercise with a predic… · 2020

Supervised Fine-Tuning

Continues next-token training on curated demonstrations of the behaviour wanted. Cheap, stable, and bounded by how well the desired behaviour can be… · 2021

Chinchilla Scaling

Refits the compute-optimal frontier and finds parameters and tokens should grow together in roughly equal proportion. Implied the models of the day w… · 2022

Reward Model

A model trained on pairwise human preferences to score responses, standing in for the objective nobody can write down. Also the component that gets g… · 2022

RLHF

Fits a reward model to human preference comparisons, then optimises the language model against it. The step that turned a next-token predictor into s… · 2022

Direct Preference Optimization

Shows the preference objective can be optimised as a classification loss on the model itself, with no reward model and no rollouts. · 2023

Reinforcement Learning from Verifiable Rewards

Replaces the learned reward with a programmatic check — does the proof verify, do the tests pass, is the answer right. Works only where correctness i… · 2025

Autoregressive Language Model

A decoder-only transformer trained to predict the next token. The objective is trivial and the capability that falls out of scaling it was not antici…