A Tighter Convergence Proof of Reverse Experience Replay

TL;DR

This paper provides a refined theoretical analysis of Reverse Experience Replay in reinforcement learning, demonstrating its convergence with larger learning rates and longer sequences, thus bridging previous gaps between theory and practice.

Contribution

It offers a tighter convergence proof for RER, allowing for larger learning rates and longer sequences than prior analyses.

Findings

RER converges with larger learning rates.

RER remains effective with longer sequences.

Theoretical analysis aligns better with empirical observations.

Abstract

In reinforcement learning, Reverse Experience Replay (RER) is a recently proposed algorithm that attains better sample complexity than the classic experience replay method. RER requires the learning algorithm to update the parameters through consecutive state-action-reward tuples in reverse order. However, the most recent theoretical analysis only holds for a minimal learning rate and short consecutive steps, which converge slower than those large learning rate algorithms without RER. In view of this theoretical and empirical gap, we provide a tighter analysis that mitigates the limitation on the learning rate and the length of consecutive steps. Furthermore, we show theoretically that RER converges with a larger learning rate and a longer sequence.