Efficient Parallel Methods for Deep Reinforcement Learning

TL;DR

This paper introduces a GPU-friendly, parallel framework for deep reinforcement learning that accelerates training times and is compatible with various algorithms, demonstrated by achieving state-of-the-art results on Atari games.

Contribution

The authors present a novel, algorithm-agnostic parallelization framework for deep reinforcement learning that significantly reduces training time on a single machine.

Findings

Achieved state-of-the-art Atari performance within hours

Framework is compatible with multiple RL algorithms

Open-source implementation available for rapid experimentation

Abstract

We propose a novel framework for efficient parallelization of deep reinforcement learning algorithms, enabling these algorithms to learn from multiple actors on a single machine. The framework is algorithm agnostic and can be applied to on-policy, off-policy, value based and policy gradient based algorithms. Given its inherent parallelism, the framework can be efficiently implemented on a GPU, allowing the usage of powerful models while significantly reducing training time. We demonstrate the effectiveness of our framework by implementing an advantage actor-critic algorithm on a GPU, using on-policy experiences and employing synchronous updates. Our algorithm achieves state-of-the-art performance on the Atari domain after only a few hours of training. Our framework thus opens the door for much faster experimentation on demanding problem domains. Our implementation is open-source and is made public at https://github.com/alfredvc/paac