Fully Sharded Data Parallel

AI and Machine Learning · Distributed Training · 2023 · also: FSDP · fully-sharded-data-parallel.yaml

The sharded-everything form of ZeRO as a framework primitive, gathering each layer's parameters just before use and releasing them straight after.

• supersedescorrects · extends
• classifiesspecializes · part-of
• substitutes forapproximates · alternative-to
• depends onrequires · validates

Colour is the family; a dashed line is the second member of it.

This node

• specializesis a specific case of ZeRO Shardingshards parameters as well as optimizer state, per layer

References

• PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel — Zhao et al. — 2023 · link