Splits individual weight matrices across devices so one layer is computed jointly. Communication lands inside every layer, so it wants the fastest interconnect available.
supersedescorrects · extends
classifiesspecializes · part-of
substitutes forapproximates · alternative-to
depends onrequires · validates
Colour is the family; a dashed line is the second member of it.
Drag to pan · scroll to zoom · click a node to open it
This node
correctsfixes a defect in Data Parallelismone replica of the model must still fit on one device
Referenced by
alternative-toPipeline Parallelism is a competing approach to thissplits between layers rather than inside one
References
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism — Shoeybi et al. — 2019 · link