Weight Decay

AI and Machine Learning · Training and Optimisation · weight-decay.yaml

Shrinks weights toward zero every step. The oldest regulariser, and the one whose interaction with adaptive optimisers turned out to be subtler than anyone assumed.

• supersedescorrects · extends
• classifiesspecializes · part-of
• substitutes forapproximates · alternative-to
• depends onrequires · validates

Colour is the family; a dashed line is the second member of it.

This node

• alternative-tois a competing approach to Dropoutconstrains weight magnitude rather than co-adaptation

Referenced by

• alternative-toEarly Stopping is a competing approach to thisstops before overfitting rather than penalising capacity

References

• Background — Ridge regression on Wikipedia