Accelerated Methods for Deep Reinforcement Learning

TL;DR

This paper presents a unified framework for accelerating deep reinforcement learning by optimizing algorithms for modern CPU-GPU architectures, enabling faster experiments and training times.

Contribution

It introduces a parallelization framework that significantly speeds up deep RL training on CPUs and GPUs, allowing complex tasks like Atari game learning in minutes.

Findings

Parallelization improves training speed without performance loss

Large batch sizes do not harm sample efficiency

Training on DGX-1 achieves Atari strategies in minutes

Abstract

Deep reinforcement learning (RL) has achieved many recent successes, yet experiment turn-around time remains a key bottleneck in research and in practice. We investigate how to optimize existing deep RL algorithms for modern computers, specifically for a combination of CPUs and GPUs. We confirm that both policy gradient and Q-value learning algorithms can be adapted to learn using many parallel simulator instances. We further find it possible to train using batch sizes considerably larger than are standard, without negatively affecting sample complexity or final performance. We leverage these facts to build a unified framework for parallelization that dramatically hastens experiments in both classes of algorithm. All neural network computations use GPUs, accelerating both data collection and training. Our results include using an entire DGX-1 to learn successful strategies in Atari games in mere minutes, using both synchronous and asynchronous algorithms.