Transformer

AI and Machine Learning · Sequence Models · 2017 · transformer.yaml

Drops recurrence entirely and stacks self-attention with feed-forward layers. Because no step waits on the previous one, the whole sequence trains in parallel, which is what let model size scale.

Colour is the family; a dashed line is the second member of it.

Drag to pan · scroll to zoom · click a node to open it
G attention Attention autoregressive-language-model Autoregressive Language Model transformer Transformer autoregressive-language-model->transformer decoder only, predicting forward with a causal mask bert BERT bert->transformer encoder only, trained by masking rather than predicting kv-cache KV Cache kv-cache->transformer generation would recompute the prefix per token layer-normalization Layer Normalization lora LoRA lora->transformer adapts a frozen model through small low-rank updates lstm LSTM mixture-of-experts Mixture of Experts mixture-of-experts->transformer compute grows with capacity if every weight is used positional-encoding Positional Encoding positional-encoding->transformer transformer->attention it is the only operation left that mixes positions transformer->layer-normalization the residual stack does not converge without it transformer->lstm recurrence forbids parallelism across sequence positions

This node

Referenced by

References