Scaling Laws

AI and Machine Learning · Language Models · 2020 · scaling-laws.yaml

Loss falls as a power law in parameters, data and compute, over many orders of magnitude. Turned model design into a budgeting exercise with a predictable return.

Colour is the family; a dashed line is the second member of it.

Drag to pan · scroll to zoom · click a node to open it
G autoregressive-language-model Autoregressive Language Model chinchilla-scaling Chinchilla Scaling scaling-laws Scaling Laws chinchilla-scaling->scaling-laws the earlier fits left models too large for their data scaling-laws->autoregressive-language-model predicts the loss a given budget will reach

This node

Referenced by

References