The Cross-environment Hyperparameter Setting Benchmark for Reinforcement Learning

TL;DR

This paper proposes a new benchmark for evaluating reinforcement learning algorithms across multiple environments using a single hyperparameter setting, emphasizing robustness and low computational cost, and demonstrates its effectiveness through empirical studies.

Contribution

The paper introduces the Cross-environment Hyperparameter Setting Benchmark, a novel empirical methodology for robust, low-cost evaluation of RL algorithms across diverse environments.

Findings

The benchmark is robust to statistical noise and consistent across repeated tests.

It is computationally inexpensive, enabling low-cost, statistically sound insights.

Empirical study shows no significant difference between noise types in DDPG exploration.

Abstract

This paper introduces a new empirical methodology, the Cross-environment Hyperparameter Setting Benchmark, that compares RL algorithms across environments using a single hyperparameter setting, encouraging algorithmic development which is insensitive to hyperparameters. We demonstrate that this benchmark is robust to statistical noise and obtains qualitatively similar results across repeated applications, even when using few samples. This robustness makes the benchmark computationally cheap to apply, allowing statistically sound insights at low cost. We demonstrate two example instantiations of the CHS, on a set of six small control environments (SC-CHS) and on the entire DM Control suite of 28 environments (DMC-CHS). Finally, to illustrate the applicability of the CHS to modern RL algorithms on challenging environments, we conduct a novel empirical study of an open question in the continuous control literature. We show, with high confidence, that there is no meaningful difference in performance between Ornstein-Uhlenbeck noise and uncorrelated Gaussian noise for exploration with the DDPG algorithm on the DMC-CHS.