Proximal Policy Optimization

AI and Machine Learning · Reinforcement Learning · 2017 · also: PPO · ppo.yaml

Clips the ratio between new and old policy so an update cannot move too far from the data it was collected under. Most of TRPO's stability with none of its second-order machinery.

• supersedescorrects · extends
• classifiesspecializes · part-of
• substitutes forapproximates · alternative-to
• depends onrequires · validates

Colour is the family; a dashed line is the second member of it.

This node

• correctsfixes a defect in Policy Gradientan unconstrained step can collapse the policy
• requiresdoes not work without Importance Samplingthe ratio it clips is the importance weight of reused data
• approximatesis a cheaper stand-in for Trust Region Policy Optimizationclipping instead of a constrained second-order solve

Referenced by

• correctsGroup Relative Policy Optimization fixes a defect in thisthe critic is a second model the size of the policy
• requiresRLHF does not work without thisthe reward model is not differentiable through text

References

• Proximal Policy Optimization Algorithms — Schulman, Wolski, Dhariwal, Radford, Klimov — 2017 · link
• Trust Region Policy Optimization — Schulman, Levine, Abbeel, Jordan, Moritz — 2015 · link