AI and Machine Learning

Sequence Models

Carrying information across a sequence, and the successive removal of bottlenecks.

Colour is the family; a dashed line is the second member of it.

Drag to pan · scroll to zoom · click a node to open it
G attention Attention recurrent-network Recurrent Network attention->recurrent-network a fixed-size hidden state cannot carry a whole sequence flash-attention FlashAttention flash-attention->attention it is memory-bandwidth bound, not compute bound cache-hierarchy Cache Hierarchy flash-attention->cache-hierarchy it tiles attention to fit in on-chip memory lstm LSTM lstm->recurrent-network gradients vanish or explode over long sequences positional-encoding Positional Encoding transformer Transformer positional-encoding->transformer transformer->attention it is the only operation left that mixes positions transformer->lstm recurrence forbids parallelism across sequence positions layer-normalization Layer Normalization transformer->layer-normalization the residual stack does not converge without it

6 nodes

LSTM

Adds a cell state with multiplicative gates controlling what is written, kept and read. The gates give the gradient a route through time that neither… · 1997

Attention

Compute a weighted sum over all positions, with weights from a learned compatibility score. Every position can reach every other in one hop, regardle… · 2014

Positional Encoding

Injects each token's position into its representation, since attention alone sees an unordered set. Sinusoidal at first, later usually learned or app… · 2017

Transformer

Drops recurrence entirely and stacks self-attention with feed-forward layers. Because no step waits on the previous one, the whole sequence trains in… · 2017

FlashAttention

Tiles attention so the score matrix is never written to memory, recomputing it in fast on-chip storage instead. Exact, not approximate: the same resu… · 2022

Recurrent Network

Carry a hidden state forward one step at a time. The natural shape for sequences, and the source of every problem the rest of this cluster fixes.