Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reuse

TL;DR

This paper introduces Sample Multiple Reuse (SMR), a method that enhances sample efficiency in off-policy reinforcement learning for continuous control by reusing fixed samples multiple times within a single update, improving performance without extra tuning.

Contribution

The paper proposes SMR, a novel approach for off-policy RL that theoretically guarantees convergence and empirically improves sample efficiency across various benchmarks.

Findings

SMR significantly improves sample efficiency of baseline algorithms.

SMR achieves better performance without additional hyperparameter tuning.

Theoretical analysis confirms convergence properties of SMR.

Abstract

Sample efficiency is one of the most critical issues for online reinforcement learning (RL). Existing methods achieve higher sample efficiency by adopting model-based methods, Q-ensemble, or better exploration mechanisms. We, instead, propose to train an off-policy RL agent via updating on a fixed sampled batch multiple times, thus reusing these samples and better exploiting them within a single optimization loop. We name our method sample multiple reuse (SMR). We theoretically show the properties of Q-learning with SMR, e.g., convergence. Furthermore, we incorporate SMR with off-the-shelf off-policy RL algorithms and conduct experiments on a variety of continuous control benchmarks. Empirical results show that SMR significantly boosts the sample efficiency of the base methods across most of the evaluated tasks without any hyperparameter tuning or additional tricks.