AI and Machine Learning

Efficiency and Deployment

Fitting training and inference into the memory and bandwidth actually available.

Colour is the family; a dashed line is the second member of it.

Drag to pan · scroll to zoom · click a node to open it
G gradient-checkpointing Gradient Checkpointing backpropagation Backpropagation gradient-checkpointing->backpropagation keeping every activation exhausts memory knowledge-distillation Knowledge Distillation cross-entropy-loss Cross-Entropy Loss knowledge-distillation->cross-entropy-loss the target is the teacher's whole distribution kv-cache KV Cache transformer Transformer kv-cache->transformer generation would recompute the prefix per token lora LoRA lora->transformer adapts a frozen model through small low-rank updates mixed-precision-training Mixed-Precision Training half-precision Half Precision mixed-precision-training->half-precision the halved bandwidth is the entire point stochastic-gradient-descent Stochastic Gradient Descent mixed-precision-training->stochastic-gradient-descent full-precision training is bandwidth bound mixture-of-experts Mixture of Experts mixture-of-experts->transformer compute grows with capacity if every weight is used quantization Quantization floating-point Floating-Point Arithmetic quantization->floating-point integers plus a scale, trading range for footprint speculative-decoding Speculative Decoding speculative-decoding->kv-cache decoding stays serial, one token per pass

8 nodes

Knowledge Distillation

Trains a small model against the large model's full output distribution rather than the hard labels. The relative probabilities of the wrong answers… · 2015

Gradient Checkpointing

Keeps only a subset of activations and recomputes the rest during the backward pass, trading arithmetic for memory at roughly the square root of the… · 2016

Mixed-Precision Training

Stores and multiplies in sixteen bits while keeping a master copy of the weights and the accumulations in thirty-two, with a loss scale to keep small… · 2017

Mixture of Experts

Routes each token to a few of many expert subnetworks, so parameter count grows without the arithmetic growing with it. The routing is the hard part,… · 2017

LoRA

Freezes the pretrained weights and learns a low-rank update beside them. Fine-tuning then costs a fraction of the memory, and the adapters can be swa… · 2021

Quantization

Represents weights, and sometimes activations, as low-bit integers with a scale. Most of the difficulty is the handful of outlier channels whose rang… · 2022

Speculative Decoding

A small draft model proposes several tokens and the large model verifies them in one batched pass, keeping the longest correct prefix. Exact: the out… · 2022

KV Cache

Stores the keys and values already computed for the prefix so each generated token attends without recomputing them. Turns quadratic regeneration int…