Keeps only a subset of activations and recomputes the rest during the backward pass, trading arithmetic for memory at roughly the square root of the naive cost.
supersedescorrects · extends
classifiesspecializes · part-of
substitutes forapproximates · alternative-to
depends onrequires · validates
Colour is the family; a dashed line is the second member of it.
Drag to pan · scroll to zoom · click a node to open it
This node
correctsfixes a defect in Backpropagationkeeping every activation exhausts memory
Referenced by
alternative-toZeRO Sharding is a competing approach to thisshards state across devices instead of recomputing it
References
Training Deep Nets with Sublinear Memory Cost — Chen, Xu, Zhang, Guestrin — 2016 · link