Group Relative Policy Optimization

AI and Machine Learning · Reinforcement Learning · 2024 · also: GRPO · grpo.yaml

Samples a group of completions per prompt and uses their mean reward as the baseline, so the advantage is relative within the group. Drops the value network entirely.

Colour is the family; a dashed line is the second member of it.

Drag to pan · scroll to zoom · click a node to open it
G actor-critic Actor-Critic grpo Group Relative Policy Optimization grpo->actor-critic a group-relative baseline instead of a learned one ppo Proximal Policy Optimization grpo->ppo the critic is a second model the size of the policy verifiable-rewards Reinforcement Learning from Verifiable Rewards verifiable-rewards->grpo one scalar per completion, with no critic to fit

This node

Referenced by

References