Shared Foundations

Execution Hardware

What the machine is actually good at, and which of its limits a technique is fighting.

Colour is the family; a dashed line is the second member of it.

Drag to pan · scroll to zoom · click a node to open it
G arithmetic-intensity Arithmetic Intensity roofline-model Roofline Model arithmetic-intensity->roofline-model branch-divergence Branch Divergence simt SIMT branch-divergence->simt cache-hierarchy Cache Hierarchy memory-coalescing Memory Coalescing memory-coalescing->simt the lanes must issue their loads in the same cycle simd SIMD simt->simd lanes get their own program counter, so branches compile

7 nodes

Roofline Model

Plots attainable performance against arithmetic intensity, bounded by peak compute on one side and memory bandwidth on the other. Tells you which of… · 2009

Arithmetic Intensity

Operations performed per byte moved. The single number that decides whether a kernel is compute bound or bandwidth bound, and therefore which optimis…

Branch Divergence

When lanes in a group take different sides of a branch, the hardware runs both sides and masks off the lanes that should not execute. Cost is the sum…

Cache Hierarchy

Successive levels trade capacity for latency and bandwidth. Almost every optimisation below the algorithmic level is a rearrangement to make a workin…

Memory Coalescing

Lanes whose addresses fall in one cache line are served by a single transaction. Scattered addresses become one transaction each, which is the differ…

SIMD

One instruction applied to many data lanes at once. The arrangement that makes throughput hardware cheap: fetch and decode are paid once for a whole…

SIMT

Lanes are presented as independent threads sharing an instruction stream. Lets ordinary branching code be written for vector hardware, at the cost of…