Reward Model

AI and Machine Learning · Language Models · 2022 · reward-model.yaml

A model trained on pairwise human preferences to score responses, standing in for the objective nobody can write down. Also the component that gets gamed once the policy optimises against it hard enough.

• supersedescorrects · extends
• classifiesspecializes · part-of
• substitutes forapproximates · alternative-to
• depends onrequires · validates

Colour is the family; a dashed line is the second member of it.

This node

• part-ofis a subtopic of RLHF

References

• Training language models to follow instructions with human feedback — Ouyang et al. — 2022 · link