Mixed-Precision Training

AI and Machine Learning · Efficiency and Deployment · 2017 · mixed-precision-training.yaml

Stores and multiplies in sixteen bits while keeping a master copy of the weights and the accumulations in thirty-two, with a loss scale to keep small gradients representable.

• supersedescorrects · extends
• classifiesspecializes · part-of
• substitutes forapproximates · alternative-to
• depends onrequires · validates

Colour is the family; a dashed line is the second member of it.

This node

• requiresdoes not work without Half Precisionthe halved bandwidth is the entire point
• correctsfixes a defect in Stochastic Gradient Descentfull-precision training is bandwidth bound

References

• Mixed Precision Training — Micikevicius et al. — 2017 · link