Regularized Q-learning through Robust Averaging

TL;DR

This paper introduces 2RA Q-learning, a novel algorithm that uses distributionally robust estimation to control bias, ensuring convergence and improved performance over existing Q-learning methods.

Contribution

The paper presents a new distributionally robust estimator for Q-learning that allows explicit bias control and maintains computational efficiency, with proven convergence and superior empirical results.

Findings

2RA Q-learning converges to the optimal policy in tabular settings.

The estimator effectively controls estimation bias.

Numerical experiments show improved performance over existing methods.

Abstract

We propose a new Q-learning variant, called 2RA Q-learning, that addresses some weaknesses of existing Q-learning methods in a principled manner. One such weakness is an underlying estimation bias which cannot be controlled and often results in poor performance. We propose a distributionally robust estimator for the maximum expected value term, which allows us to precisely control the level of estimation bias introduced. The distributionally robust estimator admits a closed-form solution such that the proposed algorithm has a computational cost per iteration comparable to Watkins' Q-learning. For the tabular case, we show that 2RA Q-learning converges to the optimal policy and analyze its asymptotic mean-squared error. Lastly, we conduct numerical experiments for various settings, which corroborate our theoretical findings and indicate that 2RA Q-learning often performs better than existing methods.