Learning-Rate Warmup

AI and Machine Learning · Training and Optimisation · 2017 · learning-rate-warmup.yaml

Ramps the step size up from near zero over the first few thousand updates instead of starting at full rate.

• supersedescorrects · extends
• classifiesspecializes · part-of
• substitutes forapproximates · alternative-to
• depends onrequires · validates

Colour is the family; a dashed line is the second member of it.

This node

• correctsfixes a defect in Adamearly steps are huge while the second moment is noisy

Referenced by

• extendsCosine Learning-Rate Schedule adds capability to thisanneals after the ramp instead of holding the rate
• requiresLarge-Batch Training does not work without thisthe scaled rate is unstable over the first epochs

References

• Attention Is All You Need — Vaswani et al. — 2017 · link