IMPALA

AI and Machine Learning · Reinforcement Learning · 2018 · also: Importance Weighted Actor-Learner Architecture, V-trace · impala.yaml

Actors ship trajectories to a central learner instead of gradients, so they never wait on a parameter sync, and V-trace corrects for how far the policy has moved by the time a batch is used. The shape distributed RL settled into.

• supersedescorrects · extends
• classifiesspecializes · part-of
• substitutes forapproximates · alternative-to
• depends onrequires · validates

Colour is the family; a dashed line is the second member of it.

This node

• correctsfixes a defect in Asynchronous Advantage Actor-Critican actor that computes gradients stalls at every parameter sync
• requiresdoes not work without Importance Samplingthe trajectory is off-policy by the time the learner gets to it

References

• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures — Espeholt et al. — 2018 · link