Addressing Maximization Bias in Reinforcement Learning with Two-Sample Testing

TL;DR

This paper introduces new statistical estimators, the T-Estimator and K-Estimator, to address maximization bias in reinforcement learning, improving value estimation accuracy and stability across various tasks.

Contribution

The paper proposes the T-Estimator and K-Estimator, novel bias control methods for reinforcement learning, with convergence guarantees and adaptive variants for better performance.

Findings

TE and KE effectively control bias in RL value estimates

Adaptive TE reduces estimation bias dynamically

Algorithms with TE and KE outperform traditional methods

Abstract

Value-based reinforcement-learning algorithms have shown strong results in games, robotics, and other real-world applications. Overestimation bias is a known threat to those algorithms and can sometimes lead to dramatic performance decreases or even complete algorithmic failure. We frame the bias problem statistically and consider it an instance of estimating the maximum expected value (MEV) of a set of random variables. We propose the $T$-Estimator (TE) based on two-sample testing for the mean, that flexibly interpolates between over- and underestimation by adjusting the significance level of the underlying hypothesis tests. We also introduce a generalization, termed $K$-Estimator (KE), that obeys the same bias and variance bounds as the TE and relies on a nearly arbitrary kernel function. We introduce modifications of $Q$-Learning and the Bootstrapped Deep $Q$-Network (BDQN) using the TE and the KE, and prove convergence in the tabular setting. Furthermore, we propose an adaptive variant of the TE-based BDQN that dynamically adjusts the significance level to minimize the absolute estimation bias. All proposed estimators and algorithms are thoroughly tested and validated on diverse tasks and environments, illustrating the bias control and performance potential of the TE and KE.