Distributed Prioritized Experience Replay

TL;DR

This paper introduces a distributed deep reinforcement learning architecture that leverages prioritized experience replay to efficiently learn from large-scale data, significantly improving performance and training speed on complex environments.

Contribution

It presents a scalable distributed architecture that decouples acting from learning, utilizing prioritized experience replay to enhance data efficiency and performance.

Findings

Achieved better final performance on Arcade Learning Environment

Reduced training time significantly compared to previous methods

Effectively scaled reinforcement learning to larger data sets

Abstract

We propose a distributed architecture for deep reinforcement learning at scale, that enables agents to learn effectively from orders of magnitude more data than previously possible. The algorithm decouples acting from learning: the actors interact with their own instances of the environment by selecting actions according to a shared neural network, and accumulate the resulting experience in a shared experience replay memory; the learner replays samples of experience and updates the neural network. The architecture relies on prioritized experience replay to focus only on the most significant data generated by the actors. Our architecture substantially improves the state of the art on the Arcade Learning Environment, achieving better final performance in a fraction of the wall-clock training time.