KV Cache

AI and Machine Learning · Efficiency and Deployment · kv-cache.yaml

Stores the keys and values already computed for the prefix so each generated token attends without recomputing them. Turns quadratic regeneration into linear, and makes memory the binding constraint.

• supersedescorrects · extends
• classifiesspecializes · part-of
• substitutes forapproximates · alternative-to
• depends onrequires · validates

Colour is the family; a dashed line is the second member of it.

This node

• correctsfixes a defect in Transformergeneration would recompute the prefix per token

Referenced by

• correctsSpeculative Decoding fixes a defect in thisdecoding stays serial, one token per pass

References

• Background — Large language model on Wikipedia