Doubles the initial variance to account for a rectifier discarding half its input. The correction that made very deep rectified networks converge from scratch.
supersedescorrects · extends
classifiesspecializes · part-of
substitutes forapproximates · alternative-to
depends onrequires · validates
Colour is the family; a dashed line is the second member of it.
Drag to pan · scroll to zoom · click a node to open it
This node
correctsfixes a defect in Weight Initializationthe usual variance assumes a symmetric activation
requiresdoes not work without ReLUthe factor of two is exactly what the rectifier discards
References
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification — He, Zhang, Ren, Sun — 2015 · link