ReLU

AI and Machine Learning · Training and Optimisation · also: Rectified linear unit · relu.yaml

Passes positives through unchanged and zeroes negatives. The derivative is exactly one on the active half, so nothing attenuates the gradient there.

• supersedescorrects · extends
• classifiesspecializes · part-of
• substitutes forapproximates · alternative-to
• depends onrequires · validates

Colour is the family; a dashed line is the second member of it.

This node

• correctsfixes a defect in Backpropagationsaturating activations drive the gradient toward zero

Referenced by

• requiresHe Initialization does not work without thisthe factor of two is exactly what the rectifier discards

References

• Background — Rectified linear unit on Wikipedia