Partitions optimizer state, gradients and parameters across data- parallel ranks instead of replicating them, gathering each shard only while it is needed.
supersedescorrects · extends
classifiesspecializes · part-of
substitutes forapproximates · alternative-to
depends onrequires · validates
Colour is the family; a dashed line is the second member of it.
Drag to pan · scroll to zoom · click a node to open it
This node
correctsfixes a defect in Data Parallelismevery replica redundantly stores the same optimizer state
alternative-tois a competing approach to Gradient Checkpointingshards state across devices instead of recomputing it
Referenced by
specializesFully Sharded Data Parallel is a specific case of thisshards parameters as well as optimizer state, per layer
References
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models — Rajbhandari, Rasley, Ruwase, He — 2019 · link