RLHF

AI and Machine Learning · Language Models · 2022 · also: Reinforcement learning from human feedback · rlhf.yaml

Fits a reward model to human preference comparisons, then optimises the language model against it. The step that turned a next-token predictor into something that follows instructions.

• supersedescorrects · extends
• classifiesspecializes · part-of
• substitutes forapproximates · alternative-to
• depends onrequires · validates

Colour is the family; a dashed line is the second member of it.

This node

• extendsadds capability to Autoregressive Language Modelaligns to preferences that next-token loss cannot express
• requiresdoes not work without Proximal Policy Optimizationthe reward model is not differentiable through text
• requiresdoes not work without Supervised Fine-Tuningthe policy is initialised from a supervised pass

Referenced by

• correctsDirect Preference Optimization fixes a defect in thisa reward model and an RL loop to fit a preference dataset
• part-ofReward Model is a subtopic of this
• correctsReinforcement Learning from Verifiable Rewards fixes a defect in thisa learned reward can be gamed; a checked answer cannot

References

• Training language models to follow instructions with human feedback — Ouyang et al. — 2022 · link
• Background — Reinforcement learning from human feedback on Wikipedia