Assigns consecutive layers to different devices and streams micro- batches through them. Communication is only at stage boundaries, at the cost of bubbles while the pipeline fills and drains.
supersedescorrects · extends
classifiesspecializes · part-of
substitutes forapproximates · alternative-to
depends onrequires · validates
Colour is the family; a dashed line is the second member of it.
Drag to pan · scroll to zoom · click a node to open it
This node
alternative-tois a competing approach to Tensor Parallelismsplits between layers rather than inside one
References
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism — Huang et al. — 2018 · link