Freezes the pretrained weights and learns a low-rank update beside them. Fine-tuning then costs a fraction of the memory, and the adapters can be swapped or merged afterwards.
supersedescorrects · extends
classifiesspecializes · part-of
substitutes forapproximates · alternative-to
depends onrequires · validates
Colour is the family; a dashed line is the second member of it.
Drag to pan · scroll to zoom · click a node to open it
This node
extendsadds capability to Transformeradapts a frozen model through small low-rank updates
References
LoRA: Low-Rank Adaptation of Large Language Models — Hu et al. — 2021 · link