All-Reduce

AI and Machine Learning · Distributed Training · all-reduce.yaml

A collective that sums a tensor across every participant and returns the result to all of them. Ring and tree schedules make its cost depend on interconnect bandwidth rather than device count.

• supersedescorrects · extends
• classifiesspecializes · part-of
• substitutes forapproximates · alternative-to
• depends onrequires · validates

Colour is the family; a dashed line is the second member of it.

Referenced by

• requiresData Parallelism does not work without thisgradients must be summed across replicas every step

References

• Background — Collective operation on Wikipedia