The Effects of Memory Replay in Reinforcement Learning

TL;DR

This paper models the impact of experience replay memory size and prioritization on reinforcement learning performance, revealing optimal conditions and proposing an adaptive memory algorithm validated through experiments.

Contribution

It introduces a dynamical systems model of Q-learning with experience replay, providing analytic insights into how memory size and prioritization affect learning.

Findings

Memory size significantly influences learning speed.

Prioritized replay can sometimes hinder learning.

Adaptive memory buffer algorithm improves empirical performance.

Abstract

Experience replay is a key technique behind many recent advances in deep reinforcement learning. Allowing the agent to learn from earlier memories can speed up learning and break undesirable temporal correlations. Despite its wide-spread application, very little is understood about the properties of experience replay. How does the amount of memory kept affect learning dynamics? Does it help to prioritize certain experiences? In this paper, we address these questions by formulating a dynamical systems ODE model of Q-learning with experience replay. We derive analytic solutions of the ODE for a simple setting. We show that even in this very simple setting, the amount of memory kept can substantially affect the agent's performance. Too much or too little memory both slow down learning. Moreover, we characterize regimes where prioritized replay harms the agent's learning. We show that our analytic solutions have excellent agreement with experiments. Finally, we propose a simple algorithm for adaptively changing the memory buffer size which achieves consistently good empirical performance.