ROER: Regularized Optimal Experience Replay

TL;DR

This paper introduces ROER, a novel experience replay prioritization method based on regularized occupancy optimization, which improves reinforcement learning performance by shifting replay data distribution towards on-policy optimality.

Contribution

It proposes a new TD-error-based prioritization scheme derived from regularized occupancy optimization, demonstrating improved RL performance on continuous control benchmarks.

Findings

ROER outperforms baselines in 6 out of 11 MuJoCo and DM Control tasks.

ROER achieves significant gains in offline-to-online fine-tuning, especially in challenging environments.

The method provides a theoretical foundation linking experience prioritization to occupancy distribution shifts.

Abstract

Experience replay serves as a key component in the success of online reinforcement learning (RL). Prioritized experience replay (PER) reweights experiences by the temporal difference (TD) error empirically enhancing the performance. However, few works have explored the motivation of using TD error. In this work, we provide an alternative perspective on TD-error-based reweighting. We show the connections between the experience prioritization and occupancy optimization. By using a regularized RL objective with $f-$divergence regularizer and employing its dual form, we show that an optimal solution to the objective is obtained by shifting the distribution of off-policy data in the replay buffer towards the on-policy optimal distribution using TD-error-based occupancy ratios. Our derivation results in a new pipeline of TD error prioritization. We specifically explore the KL divergence as the regularizer and obtain a new form of prioritization scheme, the regularized optimal experience replay (ROER). We evaluate the proposed prioritization scheme with the Soft Actor-Critic (SAC) algorithm in continuous control MuJoCo and DM Control benchmark tasks where our proposed scheme outperforms baselines in 6 out of 11 tasks while the results of the rest match with or do not deviate far from the baselines. Further, using pretraining, ROER achieves noticeable improvement on difficult Antmaze environment where baselines fail, showing applicability to offline-to-online fine-tuning. Code is available at \url{https://github.com/XavierChanglingLi/Regularized-Optimal-Experience-Replay}.