Stochastic Gradient Descent

AI and Machine Learning · Training and Optimisation · 1951 · also: SGD · stochastic-gradient-descent.yaml

Estimate the gradient from a small batch instead of the whole dataset. The estimate is noisy, and that noise turns out to help escape sharp minima as well as making each step affordable.

Colour is the family; a dashed line is the second member of it.

Drag to pan · scroll to zoom · click a node to open it
G batch-normalization Batch Normalization stochastic-gradient-descent Stochastic Gradient Descent batch-normalization->stochastic-gradient-descent each layer's input distribution shifts as earlier layers update gaussian-splatting 3D Gaussian Splatting gaussian-splatting->stochastic-gradient-descent the primitives are fitted to photographs by gradient descent gradient-descent Gradient Descent mixed-precision-training Mixed-Precision Training mixed-precision-training->stochastic-gradient-descent full-precision training is bandwidth bound momentum Momentum momentum->stochastic-gradient-descent plain SGD oscillates across ravines and crawls along them nerf Neural Radiance Fields nerf->stochastic-gradient-descent the field is fitted to photographs by gradient descent neural-radiance-cache Neural Radiance Cache neural-radiance-cache->stochastic-gradient-descent the cache is trained online, while rendering stochastic-gradient-descent->gradient-descent a full-batch gradient costs the entire dataset per step

This node

Referenced by

References