AdamW

AI and Machine Learning · Training and Optimisation · 2017 · adamw.yaml

Applies weight decay directly to the weights instead of adding an L2 term to the loss. The distinction is invisible in plain SGD and decisive in Adam.

• supersedescorrects · extends
• classifiesspecializes · part-of
• substitutes forapproximates · alternative-to
• depends onrequires · validates

Colour is the family; a dashed line is the second member of it.

This node

• correctsfixes a defect in Adaman L2 penalty stops being weight decay once Adam rescales it

References

• Decoupled Weight Decay Regularization — Loshchilov, Hutter — 2017 · link