Stratified Experience Replay: Correcting Multiplicity Bias in Off-Policy Reinforcement Learning

TL;DR

This paper identifies a bias in experience replay sampling in deep RL, proposes a stratified sampling method to correct it, and demonstrates improved training stability and performance.

Contribution

It introduces a novel stratified sampling scheme to correct multiplicity bias in off-policy reinforcement learning, challenging the conventional uniform sampling approach.

Findings

Stratified sampling reduces bias in experience replay.

Correcting sampling bias improves RL training stability.

Proposed method outperforms uniform sampling in experiments.

Abstract

Deep Reinforcement Learning (RL) methods rely on experience replay to approximate the minibatched supervised learning setting; however, unlike supervised learning where access to lots of training data is crucial to generalization, replay-based deep RL appears to struggle in the presence of extraneous data. Recent works have shown that the performance of Deep Q-Network (DQN) degrades when its replay memory becomes too large. This suggests that outdated experiences somehow impact the performance of deep RL, which should not be the case for off-policy methods like DQN. Consequently, we re-examine the motivation for sampling uniformly over a replay memory, and find that it may be flawed when using function approximation. We show that -- despite conventional wisdom -- sampling from the uniform distribution does not yield uncorrelated training samples and therefore biases gradients during training. Our theory prescribes a special non-uniform distribution to cancel this effect, and we propose a stratified sampling scheme to efficiently implement it.