Layer Normalization

AI and Machine Learning · Training and Optimisation · 2016 · also: LayerNorm · layer-normalization.yaml

Normalise across the features of a single example rather than across the batch, so the computation for one input never depends on which other inputs it was batched with.

• supersedescorrects · extends
• classifiesspecializes · part-of
• substitutes forapproximates · alternative-to
• depends onrequires · validates

Colour is the family; a dashed line is the second member of it.

This node

• correctsfixes a defect in Batch Normalizationbatch statistics make training and inference disagree

Referenced by

• correctsRMS Normalization fixes a defect in thisre-centring costs time and contributes nothing measurable
• requiresTransformer does not work without thisthe residual stack does not converge without it

References

• Layer Normalization — Ba, Kiros, Hinton — 2016 · link
• Background — Normalization (machine learning) on Wikipedia