Replay across Experiments: A Natural Extension of Off-Policy RL

TL;DR

This paper introduces Replay Across Experiments (RaE), a simple framework that reuses experience data across multiple RL experiments to improve efficiency, exploration, and robustness in various challenging control tasks.

Contribution

The paper presents a novel, minimal-adaptation method for reusing experience across experiments, enhancing RL performance and research efficiency.

Findings

RaE improves controller performance across multiple RL algorithms.

Reusing experience enhances exploration in challenging tasks.

RaE demonstrates robustness to data quality and hyperparameter variations.

Abstract

Replaying data is a principal mechanism underlying the stability and data efficiency of off-policy reinforcement learning (RL). We present an effective yet simple framework to extend the use of replays across multiple experiments, minimally adapting the RL workflow for sizeable improvements in controller performance and research iteration times. At its core, Replay Across Experiments (RaE) involves reusing experience from previous experiments to improve exploration and bootstrap learning while reducing required changes to a minimum in comparison to prior work. We empirically show benefits across a number of RL algorithms and challenging control domains spanning both locomotion and manipulation, including hard exploration tasks from egocentric vision. Through comprehensive ablations, we demonstrate robustness to the quality and amount of data available and various hyperparameter choices. Finally, we discuss how our approach can be applied more broadly across research life cycles and can increase resilience by reloading data across random seeds or hyperparameter variations.