AI and Machine Learning

Reinforcement Learning

Learning from a reward signal rather than from labelled answers.

• supersedescorrects · extends
• classifiesspecializes · part-of
• substitutes forapproximates · alternative-to
• depends onrequires · validates

Colour is the family; a dashed line is the second member of it.

The 55 reasons on these edges, as text

• extendsActor-Critic adds capability to Policy Gradienta learned baseline instead of sampled returns
• specializesAsynchronous Advantage Actor-Critic is a specific case of Actor-Criticone actor and critic, updated from many environments at once
• alternative-toAsynchronous Advantage Actor-Critic is a competing approach to Experience Replayparallel environments decorrelate the updates instead of a buffer
• alternative-toBehavior Cloning is a competing approach to Policy Gradientdemonstrations instead of a reward, so nothing has to be explored
• part-ofBellman Equation is a subtopic of Markov Decision Process
• correctsConservative Q-Learning fixes a defect in Offline RLthe critic overrates actions never seen, and the policy walks straight to them
• correctsDeep Deterministic Policy Gradient fixes a defect in Deep Q-Networkthe max over actions has no closed form once an action is a vector
• specializesDeep Deterministic Policy Gradient is a specific case of Actor-Critica deterministic actor, so the critic differentiates straight into it
• requiresDeep Deterministic Policy Gradient does not work without Target Networkthe same bootstrap divergence, now in the actor as well
• correctsDeep Q-Network fixes a defect in Q-Learninga table cannot cover a high-dimensional state space
• extendsDistributional Reinforcement Learning adds capability to Q-Learningan expectation discards the shape of the returns that produced it
• correctsDouble DQN fixes a defect in Deep Q-Networkone network both picks the next action and rates it, so noise reads as value
• requiresDouble DQN does not work without Target Networkthe independent second estimator is the frozen copy already kept
• correctsDueling Network fixes a defect in Deep Q-Networkin most states the action barely matters, yet each is estimated alone
• correctsEpsilon-Greedy fixes a defect in Q-Learningacting greedily never revisits an action its first sample underrated
• part-ofEpsilon-Greedy is a subtopic of Multi-Armed Bandit
• correctsExperience Replay fixes a defect in Deep Q-Networkconsecutive transitions are correlated
• correctsGeneralized Advantage Estimation fixes a defect in Policy Gradientraw returns are too noisy to learn from directly
• requiresGeneralized Advantage Estimation does not work without Actor-Criticthe value function it leans on is the critic
• correctsGroup Relative Policy Optimization fixes a defect in Proximal Policy Optimizationthe critic is a second model the size of the policy
• alternative-toGroup Relative Policy Optimization is a competing approach to Actor-Critica group-relative baseline instead of a learned one
• correctsIMPALA fixes a defect in Asynchronous Advantage Actor-Critican actor that computes gradients stalls at every parameter sync
• requiresIMPALA does not work without Importance Samplingthe trajectory is off-policy by the time the learner gets to it
• correctsIntrinsic Motivation fixes a defect in Epsilon-Greedyrandom actions never chain into the long sequence a sparse reward needs
• extendsMaximum Entropy RL adds capability to Markov Decision Processreward alone commits to one of several equally good actions and stops looking
• specializesMulti-Armed Bandit is a specific case of Markov Decision Processa single state, so no action changes what is faced next
• alternative-toOffline RL is a competing approach to Behavior Cloningstitches better trajectories out of mediocre ones instead of copying them
• requiresOffline RL does not work without Experience Replaythe buffer is the whole world, and nothing is ever added to it
• alternative-toPolicy Gradient is a competing approach to Q-Learningoptimises the policy itself instead of a value function
• extendsPotential-Based Reward Shaping adds capability to Markov Decision Processdense guidance can be added to a sparse reward without moving its optimum
• correctsPrioritized Experience Replay fixes a defect in Experience Replayuniform sampling spends most updates on transitions with no error left
• requiresPrioritized Experience Replay does not work without Importance Samplingdrawing by error skews the expectation, and the weight puts it back
• correctsProximal Policy Optimization fixes a defect in Policy Gradientan unconstrained step can collapse the policy
• requiresProximal Policy Optimization does not work without Importance Samplingthe ratio it clips is the importance weight of reused data
• approximatesProximal Policy Optimization is a cheaper stand-in for Trust Region Policy Optimizationclipping instead of a constrained second-order solve
• specializesQ-Learning is a specific case of Temporal Difference Learningthe bootstrapped target is the best next action's value
• extendsRainbow adds capability to Deep Q-Networkevery fix had been measured alone, against the same unimproved baseline
• requiresRainbow does not work without Prioritized Experience Replayno single ablation costs more than removing it
• requiresRainbow does not work without Distributional Reinforcement Learningit pays off only late in training, where shorter papers stopped looking
• specializesREINFORCE is a specific case of Policy Gradientthe coefficient is the episode's own return, with nothing subtracted
• requiresREINFORCE does not work without Monte Carlo Integrationthe gradient is an expectation over trajectories it can only sample
• specializesSARSA is a specific case of Temporal Difference Learningthe bootstrapped target is the action the behaviour policy chose
• alternative-toSARSA is a competing approach to Q-Learningon-policy, so the risk of its own exploration is priced into the values
• specializesSoft Actor-Critic is a specific case of Maximum Entropy RLthe critic prices entropy alongside reward, so the actor stays stochastic
• correctsSoft Actor-Critic fixes a defect in Deep Deterministic Policy Gradienta deterministic actor explores only as far as the noise someone tuned for it
• alternative-toSoft Actor-Critic is a competing approach to Proximal Policy Optimizationoff-policy, so a transition is reused for the rest of training instead of dropped after one batch
• requiresSoft Actor-Critic does not work without Experience Replayits updates come from the buffer, not from the latest rollout
• correctsTarget Network fixes a defect in Deep Q-Networka target computed from the weights being updated moves with every step
• specializesTemporal Difference Learning is a specific case of Markov Decision Processestimates the value function the MDP defines
• approximatesTemporal Difference Learning is a cheaper stand-in for Bellman Equationsamples one transition where the equation averages over every next state
• correctsTrust Region Policy Optimization fixes a defect in Policy Gradienta parameter-space step moves the policy unpredictably
• correctsTwin Delayed DDPG fixes a defect in Deep Deterministic Policy Gradientthe actor is trained to maximise exactly the critic's own errors
• extendsTwin Delayed DDPG adds capability to Double DQNcarries the two-estimator fix into continuous actions, as a min over the pair
• correctsUpper Confidence Bound fixes a defect in Epsilon-Greedyuniform exploration keeps re-trying arms already measured as bad
• part-ofUpper Confidence Bound is a subtopic of Multi-Armed Bandit

34 nodes