AI and Machine Learning

Reinforcement Learning

Learning from a reward signal rather than from labelled answers.

Colour is the family; a dashed line is the second member of it.

Drag to pan · scroll to zoom · click a node to open it
G actor-critic Actor-Critic policy-gradient Policy Gradient actor-critic->policy-gradient a learned baseline instead of sampled returns deep-q-network Deep Q-Network q-learning Q-Learning deep-q-network->q-learning a table cannot cover a high-dimensional state space experience-replay Experience Replay experience-replay->deep-q-network consecutive transitions are correlated generalized-advantage-estimation Generalized Advantage Estimation generalized-advantage-estimation->actor-critic the value function it leans on is the critic generalized-advantage-estimation->policy-gradient raw returns are too noisy to learn from directly grpo Group Relative Policy Optimization grpo->actor-critic a group-relative baseline instead of a learned one ppo Proximal Policy Optimization grpo->ppo the critic is a second model the size of the policy markov-decision-process Markov Decision Process policy-gradient->q-learning optimises the policy itself instead of a value function ppo->policy-gradient an unconstrained step can collapse the policy trpo Trust Region Policy Optimization ppo->trpo clipping instead of a constrained second-order solve importance-sampling Importance Sampling ppo->importance-sampling the ratio it clips is the importance weight of reused data temporal-difference-learning Temporal Difference Learning q-learning->temporal-difference-learning the bootstrapped target is the best next action's value temporal-difference-learning->markov-decision-process estimates the value function the MDP defines trpo->policy-gradient a parameter-space step moves the policy unpredictably

11 nodes

Q-Learning

Learns the value of each state-action pair directly, converging to the optimal policy without ever modelling the transitions. · 1989

Deep Q-Network

Replaces the Q table with a convolutional network reading pixels. The result that made reinforcement learning on raw sensory input look tractable. · 2013

Experience Replay

Stores transitions in a buffer and samples them at random, so an update is not dominated by whatever just happened. · 2013

Generalized Advantage Estimation

Blends TD residuals over many horizons with an exponential weight, giving one knob that trades bias against variance in the advantage estimate. · 2015

Trust Region Policy Optimization

Constrains each update so the KL divergence between the old and new policy stays inside a trust region, giving a monotonic improvement guarantee at t… · 2015

Proximal Policy Optimization

Clips the ratio between new and old policy so an update cannot move too far from the data it was collected under. Most of TRPO's stability with none… · 2017

Group Relative Policy Optimization

Samples a group of completions per prompt and uses their mean reward as the baseline, so the advantage is relative within the group. Drops the value… · 2024

Actor-Critic

Learns a policy and a value function together, using the value estimate as the baseline the policy gradient is measured against. Most modern policy m…

Markov Decision Process

States, actions, transition probabilities and rewards, where the next state depends only on the present one. The formalism every method below is defi…

Policy Gradient

Differentiates expected return with respect to the policy parameters directly, so stochastic and continuous action spaces need no argmax over actions.

Temporal Difference Learning

Updates a value estimate toward a later estimate rather than waiting for the final return. Learns from incomplete episodes, which is what makes onlin…