Variance Reduction Based Experience Replay for Policy Optimization

TL;DR

This paper introduces VRER, a variance reduction framework for experience replay in reinforcement learning, improving policy optimization efficiency by selectively reusing informative samples and providing theoretical convergence guarantees.

Contribution

We propose VRER, a novel, theoretically grounded experience replay method that reduces variance and accelerates policy learning in reinforcement learning algorithms.

Findings

VRER accelerates policy learning across various tasks.

It improves performance over existing policy optimization methods.

Theoretical analysis confirms convergence and bias-variance trade-off.

Abstract

Effective reinforcement learning (RL) for complex stochastic systems requires leveraging historical data collected in previous iterations to accelerate policy optimization. Classical experience replay treats all past observations uniformly and fails to account for their varying contributions to learning. To overcome this limitation, we propose Variance Reduction Experience Replay (VRER), a principled framework that selectively reuses informative samples to reduce variance in policy gradient estimation. VRER is algorithm-agnostic and integrates seamlessly with existing policy optimization methods, forming the basis of our sample-efficient off-policy algorithm, Policy Gradient with VRER (PG-VRER). Motivated by the lack of rigorous theoretical analysis of experience replay, we develop a novel framework that explicitly captures dependencies introduced by Markovian dynamics and behavior-policy interactions. Using this framework, we establish finite-time convergence guarantees for PG-VRER and reveal a fundamental bias-variance trade-off: reusing older experience increases bias but simultaneously reduces gradient variance. Extensive empirical experiments demonstrate that VRER consistently accelerates policy learning and improves performance over state-of-the-art policy optimization algorithms.