Cross-Entropy Loss

AI and Machine Learning · Training and Optimisation · cross-entropy-loss.yaml

Scores a predicted distribution by the log probability it assigned to the observed outcome. The standard classification objective, and the reason logits are trained rather than probabilities.

• supersedescorrects · extends
• classifiesspecializes · part-of
• substitutes forapproximates · alternative-to
• depends onrequires · validates

Colour is the family; a dashed line is the second member of it.

This node

• requiresdoes not work without Backpropagationit is the scalar the backward pass differentiates

Referenced by

• requiresKnowledge Distillation does not work without thisthe target is the teacher's whole distribution
• correctsLabel Smoothing fixes a defect in thisa one-hot target pushes logits apart without bound

References

• Background — Cross-entropy on Wikipedia