AI and Machine Learning

Efficiency and Deployment

Fitting training and inference into the memory and bandwidth actually available.

Colour is the family; a dashed line is the second member of it.

Drag to pan · scroll to zoom · click a node to open it · hover one for the reasons on its edges, and to bring it forward
G gradient-checkpointing Gradient Checkpointing backpropagation Backpropagation gradient-checkpointing->backpropagation knowledge-distillation Knowledge Distillation cross-entropy-loss Cross-Entropy Loss knowledge-distillation->cross-entropy-loss kv-cache KV Cache transformer Transformer kv-cache->transformer lora LoRA lora->transformer mixed-precision-training Mixed-Precision Training half-precision Half Precision mixed-precision-training->half-precision stochastic-gradient-descent Stochastic Gradient Descent mixed-precision-training->stochastic-gradient-descent mixture-of-experts Mixture of Experts mixture-of-experts->transformer quantization Quantization floating-point Floating-Point Arithmetic quantization->floating-point speculative-decoding Speculative Decoding speculative-decoding->kv-cache
The 9 reasons on these edges, as text

8 nodes

Knowledge Distillation

Trains a small model against the large model's full output distribution rather than the hard labels. The relative probabilities of the wrong answers… · 2015

Gradient Checkpointing

Keeps only a subset of activations and recomputes the rest during the backward pass, trading arithmetic for memory at roughly the square root of the… · 2016

Mixed-Precision Training

Stores and multiplies in sixteen bits while keeping a master copy of the weights and the accumulations in thirty-two, with a loss scale to keep small… · 2017

Mixture of Experts

Routes each token to a few of many expert subnetworks, so parameter count grows without the arithmetic growing with it. The routing is the hard part,… · 2017

LoRA

Freezes the pretrained weights and learns a low-rank update beside them. Fine-tuning then costs a fraction of the memory, and the adapters can be swa… · 2021

Quantization

Represents weights, and sometimes activations, as low-bit integers with a scale. Most of the difficulty is the handful of outlier channels whose rang… · 2022

Speculative Decoding

A small draft model proposes several tokens and the large model verifies them in one batched pass, keeping the longest correct prefix. Exact: the out… · 2022

KV Cache

Stores the keys and values already computed for the prefix so each generated token attends without recomputing them. Turns quadratic regeneration int…