ADDQ: Adaptive Distributional Double Q-Learning

TL;DR

This paper introduces ADDQ, a simple and adaptable method built on distributional RL to reduce overestimation bias in Q-value estimation, improving convergence across various environments.

Contribution

It presents a novel, easy-to-implement framework for locally adaptive overestimation control in distributional RL algorithms, supported by theoretical analysis and empirical results.

Findings

Improved convergence in tabular, Atari, and MuJoCo environments.

Effective reduction of overestimation bias.

Easy integration with existing distributional RL algorithms.

Abstract

Bias problems in the estimation of $Q$-values are a well-known obstacle that slows down convergence of $Q$-learning and actor-critic methods. One of the reasons of the success of modern RL algorithms is partially a direct or indirect overestimation reduction mechanism. We propose an easy to implement method built on top of distributional reinforcement learning (DRL) algorithms to deal with the overestimation in a locally adaptive way. Our framework is simple to implement, existing distributional algorithms can be improved with a few lines of code. We provide theoretical evidence and use double $Q$-learning to show how to include locally adaptive overestimation control in existing algorithms. Experiments are provided for tabular, Atari, and MuJoCo environments.