Backpropagation

AI and Machine Learning · Training and Optimisation · 1986 · backpropagation.yaml

Reverse-mode differentiation over the network graph: one backward pass yields the gradient with respect to every parameter, at roughly the cost of the forward pass.

Colour is the family; a dashed line is the second member of it.

Drag to pan · scroll to zoom · click a node to open it
G backpropagation Backpropagation gradient-descent Gradient Descent backpropagation->gradient-descent it supplies the gradient the update step consumes cross-entropy-loss Cross-Entropy Loss cross-entropy-loss->backpropagation it is the scalar the backward pass differentiates dropout Dropout dropout->backpropagation units co-adapt and memorise instead of generalising gradient-checkpointing Gradient Checkpointing gradient-checkpointing->backpropagation keeping every activation exhausts memory neural-denoising Learned Denoising neural-denoising->backpropagation the filter is learned from noisy and converged pairs relu ReLU relu->backpropagation saturating activations drive the gradient toward zero residual-connection Residual Connection residual-connection->backpropagation gradients vanish through a deep stack of layers vanishing-gradient Vanishing Gradient vanishing-gradient->backpropagation

This node

Referenced by

References