R2R2: Robust Representation for Intensive Experience Reuse via Redundancy Reduction in Self-Predictive Learning

TL;DR

R2R2 introduces a regularization technique for Self-Predictive Learning in reinforcement learning, reducing overfitting and improving performance in data-scarce, high UTD regimes, validated across multiple control tasks.

Contribution

The paper proposes R2R2, a novel regularization method that addresses spectral conflicts in SPL, and demonstrates its effectiveness and orthogonality through integration with SimbaV2, setting new state-of-the-art results.

Findings

R2R2 improves TD7 performance by approximately 22% at UTD ratio of 20.

R2R2 reduces overfitting in SPL-based reinforcement learning algorithms.

Integrating SPL into SimbaV2 enhances its performance, establishing new benchmarks.

Abstract

For reinforcement learning in data-scarce domains like real-world robotics, intensive data reuse enhances efficiency but induces overfitting. While prior works focus on critic bias, representation-level instability in Self-Predictive Learning (SPL) under high Update-to-Data (UTD) regimes remains underexplored. To bridge this gap, we propose Robust Representation via Redundancy Reduction (R2R2), a regularization method within SPL. We theoretically identify that standard zero-centering conflicts with SPL's spectral properties and design a non-centered objective accordingly. We verify R2R2 on SPL-native algorithms like TD7. Furthermore, to demonstrate its orthogonality to prior advancements, we extend the state-of-the-art SimbaV2, which originally lacks SPL, by integrating a tailored SPL module, termed SimbaV2-SPL. Experiments across 11 continuous control tasks confirm that R2R2 effectively mitigates overfitting; specifically, at a UTD ratio of 20, it improves TD7 by ~22% and provides additional gains on top of SimbaV2-SPL, which itself establishes a new state-of-the-art. The code can be found at: https://github.com/songsang7/R2R2