Balanced Q-learning: Combining the Influence of Optimistic and Pessimistic Targets

TL;DR

This paper introduces Balanced Q-learning, a novel algorithm that combines optimistic and pessimistic targets to mitigate overestimation bias in Q-learning, with proven convergence and improved performance across environments.

Contribution

It proposes a new balanced target approach in Q-learning that adaptively combines optimistic and pessimistic estimates, enhancing learning robustness.

Findings

Proven convergence of the algorithm in tabular settings.

Demonstrated superior performance in various environments.

Showed that biased targets can be beneficial depending on the scenario.

Abstract

The optimistic nature of the Q-learning target leads to an overestimation bias, which is an inherent problem associated with standard $Q-$learning. Such a bias fails to account for the possibility of low returns, particularly in risky scenarios. However, the existence of biases, whether overestimation or underestimation, need not necessarily be undesirable. In this paper, we analytically examine the utility of biased learning, and show that specific types of biases may be preferable, depending on the scenario. Based on this finding, we design a novel reinforcement learning algorithm, Balanced Q-learning, in which the target is modified to be a convex combination of a pessimistic and an optimistic term, whose associated weights are determined online, analytically. We prove the convergence of this algorithm in a tabular setting, and empirically demonstrate its superior learning performance in various environments.