Increasing the Action Gap: New Operators for Reinforcement Learning

TL;DR

This paper proposes new operators for Q-learning that increase the action gap to improve policy robustness, demonstrating their effectiveness through theoretical analysis and empirical results on Atari games.

Contribution

Introduction of a consistent Bellman operator that increases the action gap while preserving optimality, applicable to both tabular and continuous problems.

Findings

The consistent Bellman operator increases the action gap at each state.

Empirical results show superior performance on Atari 2600 games.

Theoretical conditions for operators to preserve optimality are established.

Abstract

This paper introduces new optimality-preserving operators on Q-functions. We first describe an operator for tabular representations, the consistent Bellman operator, which incorporates a notion of local policy consistency. We show that this local consistency leads to an increase in the action gap at each state; increasing this gap, we argue, mitigates the undesirable effects of approximation and estimation errors on the induced greedy policies. This operator can also be applied to discretized continuous space and time problems, and we provide empirical results evidencing superior performance in this context. Extending the idea of a locally consistent operator, we then derive sufficient conditions for an operator to preserve optimality, leading to a family of operators which includes our consistent Bellman operator. As corollaries we provide a proof of optimality for Baird's advantage learning algorithm and derive other gap-increasing operators with interesting properties. We conclude with an empirical study on 60 Atari 2600 games illustrating the strong potential of these new operators.