AI and Machine Learning

Training and Optimisation

How the weights actually get set, and what each step of that lineage fixed.

Colour is the family; a dashed line is the second member of it.

Drag to pan · scroll to zoom · click a node to open it · hover one for the reasons on its edges, and to bring it forward
G adam Adam momentum Momentum adam->momentum adamw AdamW adamw->adam backpropagation Backpropagation gradient-descent Gradient Descent backpropagation->gradient-descent batch-normalization Batch Normalization stochastic-gradient-descent Stochastic Gradient Descent batch-normalization->stochastic-gradient-descent cosine-schedule Cosine Learning-Rate Schedule learning-rate-warmup Learning-Rate Warmup cosine-schedule->learning-rate-warmup cross-entropy-loss Cross-Entropy Loss cross-entropy-loss->backpropagation dropout Dropout dropout->backpropagation early-stopping Early Stopping weight-decay Weight Decay early-stopping->weight-decay gradient-clipping Gradient Clipping recurrent-network Recurrent Network gradient-clipping->recurrent-network he-initialization He Initialization relu ReLU he-initialization->relu weight-initialization Weight Initialization he-initialization->weight-initialization label-smoothing Label Smoothing label-smoothing->cross-entropy-loss large-batch-training Large-Batch Training large-batch-training->learning-rate-warmup data-parallelism Data Parallelism large-batch-training->data-parallelism layer-normalization Layer Normalization layer-normalization->batch-normalization learning-rate-warmup->adam momentum->stochastic-gradient-descent relu->backpropagation residual-connection Residual Connection residual-connection->backpropagation rms-norm RMS Normalization rms-norm->layer-normalization stochastic-gradient-descent->gradient-descent vanishing-gradient Vanishing Gradient vanishing-gradient->backpropagation weight-decay->dropout
The 23 reasons on these edges, as text

23 nodes

Stochastic Gradient Descent

Estimate the gradient from a small batch instead of the whole dataset. The estimate is noisy, and that noise turns out to help escape sharp minima as… · 1951

Momentum

Accumulate a running average of past gradients and step along that. Damps the oscillation across a narrow valley while accelerating along its floor. · 1964

Backpropagation

Reverse-mode differentiation over the network graph: one backward pass yields the gradient with respect to every parameter, at roughly the cost of th… · 1986

Weight Initialization

Chooses the starting variance so activations neither shrink nor blow up as they propagate. A deep network started wrong does not train slowly; it doe… · 2010

Gradient Clipping

Rescales the gradient when its norm exceeds a threshold. Crude, and it turns a run-ending weight update into a merely bad one. · 2012

Adam

Keeps running averages of both the gradient and its square, giving every parameter its own step size. The default optimiser almost everywhere, mostly… · 2014

Dropout

Randomly zeroes units during training, so no unit can rely on any particular other one being present. Approximates averaging over an exponential fami… · 2014

Batch Normalization

Normalise each activation across the batch, then rescale by learned parameters. Allowed much higher learning rates, which is what made very deep netw… · 2015

He Initialization

Doubles the initial variance to account for a rectifier discarding half its input. The correction that made very deep rectified networks converge fro… · 2015

Label Smoothing

Replaces the one-hot target with a slightly softened distribution, so the model is never asked to be infinitely confident. · 2015

Residual Connection

Add a layer's input to its output so the layer only has to learn a correction. Gradients then reach early layers along a path that does not attenuate… · 2015

Cosine Learning-Rate Schedule

Anneals the step size along a cosine from its peak to near zero, optionally restarting. The default decay shape, largely because it has no knobs wort… · 2016

Layer Normalization

Normalise across the features of a single example rather than across the batch, so the computation for one input never depends on which other inputs… · 2016

AdamW

Applies weight decay directly to the weights instead of adding an L2 term to the loss. The distinction is invisible in plain SGD and decisive in Adam. · 2017

Large-Batch Training

Scales the learning rate linearly with batch size so that many devices can share one step. Holds up to surprisingly large batches, and only with a wa… · 2017

Learning-Rate Warmup

Ramps the step size up from near zero over the first few thousand updates instead of starting at full rate. · 2017

RMS Normalization

Drops the mean-subtraction step and rescales by the root mean square alone. Matches LayerNorm's quality, which is the finding: the re- centring was n… · 2019

Cross-Entropy Loss

Scores a predicted distribution by the log probability it assigned to the observed outcome. The standard classification objective, and the reason log…

Early Stopping

Halts training when held-out loss stops improving. The cheapest regulariser there is, and it needs a validation split you are not otherwise using.

Gradient Descent

Step downhill along the negative gradient. Everything below is a modification of this one idea, made to survive contact with real objectives.

ReLU

Passes positives through unchanged and zeroes negatives. The derivative is exactly one on the active half, so nothing attenuates the gradient there.

Vanishing Gradient

Repeated multiplication by factors below one drives the gradient toward zero as it travels back through layers or time steps, so early parameters sto…

Weight Decay

Shrinks weights toward zero every step. The oldest regulariser, and the one whose interaction with adaptive optimisers turned out to be subtler than…