Cosine Learning-Rate Schedule

AI and Machine Learning · Training and Optimisation · 2016 · cosine-schedule.yaml

Anneals the step size along a cosine from its peak to near zero, optionally restarting. The default decay shape, largely because it has no knobs worth tuning.

• supersedescorrects · extends
• classifiesspecializes · part-of
• substitutes forapproximates · alternative-to
• depends onrequires · validates

Colour is the family; a dashed line is the second member of it.

This node

• extendsadds capability to Learning-Rate Warmupanneals after the ramp instead of holding the rate

References

• SGDR: Stochastic Gradient Descent with Warm Restarts — Loshchilov, Hutter — 2016 · link