Data Parallelism

AI and Machine Learning · Distributed Training · data-parallelism.yaml

Every device holds a full replica and processes a slice of the batch, then the gradients are summed. The simplest way to scale, and it stops working when one replica no longer fits.

• supersedescorrects · extends
• classifiesspecializes · part-of
• substitutes forapproximates · alternative-to
• depends onrequires · validates

Colour is the family; a dashed line is the second member of it.

This node

• requiresdoes not work without All-Reducegradients must be summed across replicas every step

Referenced by

• requiresLarge-Batch Training does not work without thisthe batch is large because it is split across devices
• correctsTensor Parallelism fixes a defect in thisone replica of the model must still fit on one device
• correctsZeRO Sharding fixes a defect in thisevery replica redundantly stores the same optimizer state

References

• Background — Data parallelism on Wikipedia