Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Survey

TL;DR

This survey comprehensively reviews methods for accelerating deep reinforcement learning training through parallel and distributed computing, covering architectures, synchronization, and open-source tools, and discusses future research directions.

Contribution

It provides a detailed taxonomy of existing acceleration techniques, compares open-source platforms, and highlights emerging topics and open issues in the field.

Findings

Identifies key methodologies for training acceleration.

Provides a comparative analysis of 16 open-source platforms.

Discusses future research directions in the field.

Abstract

Deep reinforcement learning has led to dramatic breakthroughs in the field of artificial intelligence for the past few years. As the amount of rollout experience data and the size of neural networks for deep reinforcement learning have grown continuously, handling the training process and reducing the time consumption using parallel and distributed computing is becoming an urgent and essential desire. In this paper, we perform a broad and thorough investigation on training acceleration methodologies for deep reinforcement learning based on parallel and distributed computing, providing a comprehensive survey in this field with state-of-the-art methods and pointers to core references. In particular, a taxonomy of literature is provided, along with a discussion of emerging topics and open issues. This incorporates learning system architectures, simulation parallelism, computing parallelism, distributed synchronization mechanisms, and deep evolutionary reinforcement learning. Further, we compare 16 current open-source libraries and platforms with criteria of facilitating rapid development. Finally, we extrapolate future directions that deserve further research.