Variance Reduction via Resampling and Experience Replay

TL;DR

This paper develops a theoretical framework for experience replay in reinforcement learning, modeling it with resampled U- and V-statistics to guarantee variance reduction and improve stability across various algorithms.

Contribution

It introduces a rigorous variance reduction framework for experience replay, extending its application to policy evaluation and kernel ridge regression, with theoretical guarantees and practical efficiency gains.

Findings

Significant variance reduction in policy evaluation tasks.

Reduced computational complexity from O(n^3) to O(n^2) in kernel methods.

Empirical validation shows improved stability and efficiency.

Abstract

Experience replay is a foundational technique in reinforcement learning that enhances learning stability by storing past experiences in a replay buffer and reusing them during training. Despite its practical success, its theoretical properties remain underexplored. In this paper, we present a theoretical framework that models experience replay using resampled $U$- and $V$-statistics, providing rigorous variance reduction guarantees. We apply this framework to policy evaluation tasks using the Least-Squares Temporal Difference (LSTD) algorithm and a Partial Differential Equation (PDE)-based model-free algorithm, demonstrating significant improvements in stability and efficiency, particularly in data-scarce scenarios. Beyond policy evaluation, we extend the framework to kernel ridge regression, showing that the experience replay-based method reduces the computational cost from the traditional $O(n^3)$ in time to as low as $O(n^2)$ in time while simultaneously reducing variance. Extensive numerical experiments validate our theoretical findings, demonstrating the broad applicability and effectiveness of experience replay in diverse machine learning tasks.