Gradient Clipping

AI and Machine Learning · Training and Optimisation · 2012 · gradient-clipping.yaml

Rescales the gradient when its norm exceeds a threshold. Crude, and it turns a run-ending weight update into a merely bad one.

• supersedescorrects · extends
• classifiesspecializes · part-of
• substitutes forapproximates · alternative-to
• depends onrequires · validates

Colour is the family; a dashed line is the second member of it.

This node

• correctsfixes a defect in Recurrent Networkexploding gradients destroy the weights in one step

References

• On the difficulty of training Recurrent Neural Networks — Pascanu, Mikolov, Bengio — 2012 · link