Momentum

AI and Machine Learning · Training and Optimisation · 1964 · momentum.yaml

Accumulate a running average of past gradients and step along that. Damps the oscillation across a narrow valley while accelerating along its floor.

• supersedescorrects · extends
• classifiesspecializes · part-of
• substitutes forapproximates · alternative-to
• depends onrequires · validates

Colour is the family; a dashed line is the second member of it.

This node

• correctsfixes a defect in Stochastic Gradient Descentplain SGD oscillates across ravines and crawls along them

Referenced by

• extendsAdam adds capability to thisadds a per-parameter step size from the second moment

References

• Background — Stochastic gradient descent on Wikipedia