Tiles attention so the score matrix is never written to memory, recomputing it in fast on-chip storage instead. Exact, not approximate: the same result, bounded by arithmetic rather than bandwidth.
supersedescorrects · extends
classifiesspecializes · part-of
substitutes forapproximates · alternative-to
depends onrequires · validates
Colour is the family; a dashed line is the second member of it.
Drag to pan · scroll to zoom · click a node to open it
This node
correctsfixes a defect in Attentionit is memory-bandwidth bound, not compute bound
requiresdoes not work without Cache Hierarchyit tiles attention to fit in on-chip memory
References
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Dao, Fu, Ermon, Rudra, Re — 2022 · link