Drops the mean-subtraction step and rescales by the root mean square alone. Matches LayerNorm's quality, which is the finding: the re- centring was never doing the work.
supersedescorrects · extends
classifiesspecializes · part-of
substitutes forapproximates · alternative-to
depends onrequires · validates
Colour is the family; a dashed line is the second member of it.
Drag to pan · scroll to zoom · click a node to open it
This node
correctsfixes a defect in Layer Normalizationre-centring costs time and contributes nothing measurable
References
Root Mean Square Layer Normalization — Zhang, Sennrich — 2019 · link