Large-Batch Training

AI and Machine Learning · Training and Optimisation · 2017 · large-batch-training.yaml

Scales the learning rate linearly with batch size so that many devices can share one step. Holds up to surprisingly large batches, and only with a warmup in front of it.

• supersedescorrects · extends
• classifiesspecializes · part-of
• substitutes forapproximates · alternative-to
• depends onrequires · validates

Colour is the family; a dashed line is the second member of it.

This node

• requiresdoes not work without Learning-Rate Warmupthe scaled rate is unstable over the first epochs
• requiresdoes not work without Data Parallelismthe batch is large because it is split across devices

References

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour — Goyal et al. — 2017 · link