Can we hop in general? A discussion of benchmark selection and design using the Hopper environment

TL;DR

This paper emphasizes the importance of careful benchmark selection in reinforcement learning, illustrating how different Hopper environment variants can significantly influence algorithm performance evaluation and calling for a more scientific approach to benchmarking.

Contribution

It highlights the need for a systematic and justified approach to benchmark selection in RL and demonstrates the impact of environment choice through a case study on Hopper variants.

Findings

Benchmark choices drastically affect performance evaluation.

Current benchmarks lack clear justification and representativeness.

A call for standardized criteria in benchmark selection.

Abstract

Empirical, benchmark-driven testing is a fundamental paradigm in the current RL community. While using off-the-shelf benchmarks in reinforcement learning (RL) research is a common practice, this choice is rarely discussed. Benchmark choices are often done based on intuitive ideas like "legged robots" or "visual observations". In this paper, we argue that benchmarking in RL needs to be treated as a scientific discipline itself. To illustrate our point, we present a case study on different variants of the Hopper environment to show that the selection of standard benchmarking suites can drastically change how we judge performance of algorithms. The field does not have a cohesive notion of what the different Hopper environments are representative - they do not even seem to be representative of each other. Our experimental results suggests a larger issue in the deep RL literature: benchmark choices are neither commonly justified, nor does there exist a language that could be used to justify the selection of certain environments. This paper concludes with a discussion of the requirements for proper discussion and evaluations of benchmarks and recommends steps to start a dialogue towards this goal.