Making Reinforcement Learning Work on Swimmer

TL;DR

This paper demonstrates that the poor performance of reinforcement learning on the SWIMMER benchmark is due to hyper-parameter tuning issues, specifically the discount factor, and provides corrected hyper-parameters for improved results.

Contribution

The paper identifies the impact of discount factor tuning on RL performance on SWIMMER and offers effective hyper-parameter settings for common RL algorithms.

Findings

Proper discount factor tuning significantly improves RL performance on SWIMMER.

Default hyper-parameters often lead to suboptimal RL results on the benchmark.

Providing tuned hyper-parameters enhances RL methods' competitiveness on SWIMMER.

Abstract

The SWIMMER environment is a standard benchmark in reinforcement learning (RL). In particular, it is often used in papers comparing or combining RL methods with direct policy search methods such as genetic algorithms or evolution strategies. A lot of these papers report poor performance on SWIMMER from RL methods and much better performance from direct policy search methods. In this technical report we show that the low performance of RL methods on SWIMMER simply comes from the inadequate tuning of an important hyper-parameter, the discount factor. Furthermore we show that, by setting this hyper-parameter to a correct value, the issue can be easily fixed. Finally, for a set of often used RL algorithms, we provide a set of successful hyper-parameters obtained with the Stable Baselines3 library and its RL Zoo.