Parallel Actors and Learners: A Framework for Generating Scalable RL Implementations

TL;DR

This paper introduces a scalable reinforcement learning framework using parallel actors and learners, optimized data structures, and memory layouts to significantly improve training efficiency on multi-core CPU systems.

Contribution

It presents a novel framework with a new prioritized replay buffer, cache-efficient data layout, and lazy writing to enhance RL training scalability on multi-core architectures.

Findings

Achieved faster RL training on CPU+GPU platforms.

Supported multiple RL algorithms like DQN and DDPG.

Reduced synchronization overheads and cache misses.

Abstract

Reinforcement Learning (RL) has achieved significant success in application domains such as robotics, games and health care. However, training RL agents is very time consuming. Current implementations exhibit poor performance due to challenges such as irregular memory accesses and thread-level synchronization overheads on CPU. In this work, we propose a framework for generating scalable reinforcement learning implementations on multi-core systems. Replay Buffer is a key component of RL algorithms which facilitates storage of samples obtained from environmental interactions and data sampling for the learning process. We define a new data structure for Prioritized Replay Buffer based on $K$-ary sum tree that supports asynchronous parallel insertions, sampling, and priority updates. To address the challenge of irregular memory accesses, we propose a novel data layout to store the nodes of the sum tree that reduces the number of cache misses. Additionally, we propose $\textit{lazy writing}$ mechanism to reduce thread-level synchronization overheads of the Replay Buffer operations. Our framework employs parallel actors to concurrently collect data via environmental interactions, and parallel learners to perform stochastic gradient descent using the collected data. Our framework supports a wide range of reinforcement learning algorithms including DQN, DDPG, etc. We demonstrate the effectiveness of our framework in accelerating RL algorithms by performing experiments on CPU + GPU platform using OpenAI benchmarks.