Never Worse, Mostly Better: Stable Policy Improvement in Deep Reinforcement Learning

TL;DR

This paper introduces EVEREST, a method that stabilizes deep reinforcement learning by making conservative policy updates, leading to more reliable improvements and reduced performance deterioration.

Contribution

The paper proposes EVEREST, a novel approach using confidence bounds for stable policy improvement in deep RL, addressing instability and variance issues in training.

Findings

EVEREST improves performance in continuous control tasks.

EVEREST stabilizes training, reducing performance drops.

Empirical results show significant gains on Atari benchmarks.

Abstract

In recent years, there has been significant progress in applying deep reinforcement learning (RL) for solving challenging problems across a wide variety of domains. Nevertheless, convergence of various methods has been shown to suffer from inconsistencies, due to algorithmic instability and variance, as well as stochasticity in the benchmark environments. Particularly, despite the fact that the agent's performance may be improving on average, it may abruptly deteriorate at late stages of training. In this work, we study methods for enhancing the agent's learning process, by providing conservative updates with respect to either the obtained history or a reference benchmark policy. Our method, termed EVEREST, obtains high confidence improvements via confidence bounds of a reference policy. Through extensive empirical analysis we demonstrate the benefit of our approach in terms of both performance and stabilization, with significant improvements in continuous control and Atari benchmarks.