Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control

TL;DR

This paper examines the reproducibility challenges of benchmarked deep reinforcement learning algorithms for continuous control, emphasizing hyper-parameter tuning, variance, and reporting standards to improve experimental consistency.

Contribution

It provides an analysis of reproducibility issues in policy gradient methods and offers guidelines for better reporting and comparison practices in continuous control tasks.

Findings

Hyper-parameters significantly affect policy gradient performance.

Variance in algorithms impacts reproducibility of results.

Guidelines improve clarity and comparability of experimental outcomes.

Abstract

Policy gradient methods in reinforcement learning have become increasingly prevalent for state-of-the-art performance in continuous control tasks. Novel methods typically benchmark against a few key algorithms such as deep deterministic policy gradients and trust region policy optimization. As such, it is important to present and use consistent baselines experiments. However, this can be difficult due to general variance in the algorithms, hyper-parameter tuning, and environment stochasticity. We investigate and discuss: the significance of hyper-parameters in policy gradients for continuous control, general variance in the algorithms, and reproducibility of reported results. We provide guidelines on reporting novel results as comparisons against baseline methods such that future researchers can make informed decisions when investigating novel methods.