Adam

AI and Machine Learning · Training and Optimisation · 2014 · adam.yaml

Keeps running averages of both the gradient and its square, giving every parameter its own step size. The default optimiser almost everywhere, mostly because it needs so little tuning.

• supersedescorrects · extends
• classifiesspecializes · part-of
• substitutes forapproximates · alternative-to
• depends onrequires · validates

Colour is the family; a dashed line is the second member of it.

This node

• extendsadds capability to Momentumadds a per-parameter step size from the second moment

Referenced by

• correctsAdamW fixes a defect in thisan L2 penalty stops being weight decay once Adam rescales it
• correctsLearning-Rate Warmup fixes a defect in thisearly steps are huge while the second moment is noisy

References

• Adam: A Method for Stochastic Optimization — Kingma, Ba — 2014 · link