Weight Initialization

AI and Machine Learning · Training and Optimisation · 2010 · weight-initialization.yaml

Chooses the starting variance so activations neither shrink nor blow up as they propagate. A deep network started wrong does not train slowly; it does not train.

• supersedescorrects · extends
• classifiesspecializes · part-of
• substitutes forapproximates · alternative-to
• depends onrequires · validates

Colour is the family; a dashed line is the second member of it.

Referenced by

• correctsHe Initialization fixes a defect in thisthe usual variance assumes a symmetric activation

References

• Background — Weight initialization on Wikipedia