Deep In-GPU Experience Replay

TL;DR

This paper introduces the first in-GPU experience replay method for reinforcement learning, significantly speeding up training by storing experience data directly on the GPU, demonstrated with a Super Smash Bros. Melee agent.

Contribution

The paper presents the novel concept of in-GPU experience replay, eliminating CPU-GPU data transfer bottlenecks during training.

Findings

In-GPU experience replay trains twice as fast as in-RAM.

Full experience replay fits on a single GPU for the Melee environment.

First implementation of in-GPU experience replay in reinforcement learning.

Abstract

Experience replay allows a reinforcement learning agent to train on samples from a large amount of the most recent experiences. A simple in-RAM experience replay stores these most recent experiences in a list in RAM, and then copies sampled batches to the GPU for training. I moved this list to the GPU, thus creating an in-GPU experience replay, and a training step that no longer has inputs copied from the CPU. I trained an agent to play Super Smash Bros. Melee, using internal game memory values as inputs and outputting controller button presses. A single state in Melee contains 27 floats, so the full experience replay fits on a single GPU. For a batch size of 128, the in-GPU experience replay trained twice as fast as the in-RAM experience replay. As far as I know, this is the first in-GPU implementation of experience replay. Finally, I note a few ideas for fitting the experience replay inside the GPU when the environment state requires more memory.