AI and Machine Learning

Training and Optimisation

How the weights actually get set, and what each step of that lineage fixed.

Colour is the family; a dashed line is the second member of it.

Drag to pan · scroll to zoom · click a node to open it
G adam Adam momentum Momentum adam->momentum adds a per-parameter step size from the second moment adamw AdamW adamw->adam an L2 penalty stops being weight decay once Adam rescales it backpropagation Backpropagation gradient-descent Gradient Descent backpropagation->gradient-descent it supplies the gradient the update step consumes batch-normalization Batch Normalization stochastic-gradient-descent Stochastic Gradient Descent batch-normalization->stochastic-gradient-descent each layer's input distribution shifts as earlier layers update cosine-schedule Cosine Learning-Rate Schedule learning-rate-warmup Learning-Rate Warmup cosine-schedule->learning-rate-warmup anneals after the ramp instead of holding the rate cross-entropy-loss Cross-Entropy Loss cross-entropy-loss->backpropagation it is the scalar the backward pass differentiates dropout Dropout dropout->backpropagation units co-adapt and memorise instead of generalising early-stopping Early Stopping weight-decay Weight Decay early-stopping->weight-decay stops before overfitting rather than penalising capacity gradient-clipping Gradient Clipping recurrent-network Recurrent Network gradient-clipping->recurrent-network exploding gradients destroy the weights in one step he-initialization He Initialization relu ReLU he-initialization->relu the factor of two is exactly what the rectifier discards weight-initialization Weight Initialization he-initialization->weight-initialization the usual variance assumes a symmetric activation label-smoothing Label Smoothing label-smoothing->cross-entropy-loss a one-hot target pushes logits apart without bound large-batch-training Large-Batch Training large-batch-training->learning-rate-warmup the scaled rate is unstable over the first epochs data-parallelism Data Parallelism large-batch-training->data-parallelism the batch is large because it is split across devices layer-normalization Layer Normalization layer-normalization->batch-normalization batch statistics make training and inference disagree learning-rate-warmup->adam early steps are huge while the second moment is noisy momentum->stochastic-gradient-descent plain SGD oscillates across ravines and crawls along them relu->backpropagation saturating activations drive the gradient toward zero residual-connection Residual Connection residual-connection->backpropagation gradients vanish through a deep stack of layers rms-norm RMS Normalization rms-norm->layer-normalization re-centring costs time and contributes nothing measurable stochastic-gradient-descent->gradient-descent a full-batch gradient costs the entire dataset per step vanishing-gradient Vanishing Gradient vanishing-gradient->backpropagation weight-decay->dropout constrains weight magnitude rather than co- adaptation

23 nodes

Stochastic Gradient Descent

Estimate the gradient from a small batch instead of the whole dataset. The estimate is noisy, and that noise turns out to help escape sharp minima as… · 1951

Momentum

Accumulate a running average of past gradients and step along that. Damps the oscillation across a narrow valley while accelerating along its floor. · 1964

Backpropagation

Reverse-mode differentiation over the network graph: one backward pass yields the gradient with respect to every parameter, at roughly the cost of th… · 1986

Weight Initialization

Chooses the starting variance so activations neither shrink nor blow up as they propagate. A deep network started wrong does not train slowly; it doe… · 2010

Gradient Clipping

Rescales the gradient when its norm exceeds a threshold. Crude, and it turns a run-ending weight update into a merely bad one. · 2012

Adam

Keeps running averages of both the gradient and its square, giving every parameter its own step size. The default optimiser almost everywhere, mostly… · 2014

Dropout

Randomly zeroes units during training, so no unit can rely on any particular other one being present. Approximates averaging over an exponential fami… · 2014

Batch Normalization

Normalise each activation across the batch, then rescale by learned parameters. Allowed much higher learning rates, which is what made very deep netw… · 2015

He Initialization

Doubles the initial variance to account for a rectifier discarding half its input. The correction that made very deep rectified networks converge fro… · 2015

Label Smoothing

Replaces the one-hot target with a slightly softened distribution, so the model is never asked to be infinitely confident. · 2015

Residual Connection

Add a layer's input to its output so the layer only has to learn a correction. Gradients then reach early layers along a path that does not attenuate… · 2015

Cosine Learning-Rate Schedule

Anneals the step size along a cosine from its peak to near zero, optionally restarting. The default decay shape, largely because it has no knobs wort… · 2016

Layer Normalization

Normalise across the features of a single example rather than across the batch, so the computation for one input never depends on which other inputs… · 2016

AdamW

Applies weight decay directly to the weights instead of adding an L2 term to the loss. The distinction is invisible in plain SGD and decisive in Adam. · 2017

Large-Batch Training

Scales the learning rate linearly with batch size so that many devices can share one step. Holds up to surprisingly large batches, and only with a wa… · 2017

Learning-Rate Warmup

Ramps the step size up from near zero over the first few thousand updates instead of starting at full rate. · 2017

RMS Normalization

Drops the mean-subtraction step and rescales by the root mean square alone. Matches LayerNorm's quality, which is the finding: the re- centring was n… · 2019

Cross-Entropy Loss

Scores a predicted distribution by the log probability it assigned to the observed outcome. The standard classification objective, and the reason log…

Early Stopping

Halts training when held-out loss stops improving. The cheapest regulariser there is, and it needs a validation split you are not otherwise using.

Gradient Descent

Step downhill along the negative gradient. Everything below is a modification of this one idea, made to survive contact with real objectives.

ReLU

Passes positives through unchanged and zeroes negatives. The derivative is exactly one on the active half, so nothing attenuates the gradient there.

Vanishing Gradient

Repeated multiplication by factors below one drives the gradient toward zero as it travels back through layers or time steps, so early parameters sto…

Weight Decay

Shrinks weights toward zero every step. The oldest regulariser, and the one whose interaction with adaptive optimisers turned out to be subtler than…