Is High Variance Unavoidable in RL? A Case Study in Continuous Control

TL;DR

This paper investigates the causes of high variance in reinforcement learning, especially in continuous control tasks, and demonstrates that simple architectural fixes like feature normalization can significantly reduce this variance.

Contribution

The study identifies numerical instability as a key cause of early variance in RL and shows that feature normalization effectively mitigates this issue, improving stability and reproducibility.

Findings

Variance mainly arises early in training due to numerical instability.

Normalizing penultimate features reduces outcome variance significantly.

Addressing instability allows for larger learning rates and more stable training.

Abstract

Reinforcement learning (RL) experiments have notoriously high variance, and minor details can have disproportionately large effects on measured outcomes. This is problematic for creating reproducible research and also serves as an obstacle for real-world applications, where safety and predictability are paramount. In this paper, we investigate causes for this perceived instability. To allow for an in-depth analysis, we focus on a specifically popular setup with high variance -- continuous control from pixels with an actor-critic agent. In this setting, we demonstrate that variance mostly arises early in training as a result of poor "outlier" runs, but that weight initialization and initial exploration are not to blame. We show that one cause for early variance is numerical instability which leads to saturating nonlinearities. We investigate several fixes to this issue and find that one particular method is surprisingly effective and simple -- normalizing penultimate features. Addressing the learning instability allows for larger learning rates, and significantly decreases the variance of outcomes. This demonstrates that the perceived variance in RL is not necessarily inherent to the problem definition and may be addressed through simple architectural modifications.