Factors of Influence of the Overestimation Bias of Q-Learning

TL;DR

This paper investigates how learning rate, discount factor, and reward signals affect overestimation bias in Q-Learning, demonstrating that parameter tuning and reward smoothing can improve value estimate accuracy.

Contribution

It identifies key parameters influencing overestimation bias and proposes a method using reward averaging to enhance Q-Learning's accuracy beyond existing approaches.

Findings

All three parameters significantly influence overestimation.

Careful tuning of parameters reduces bias.

Reward smoothing improves value estimate accuracy.

Abstract

We study whether the learning rate $\alpha$, the discount factor $\gamma$ and the reward signal $r$ have an influence on the overestimation bias of the Q-Learning algorithm. Our preliminary results in environments which are stochastic and that require the use of neural networks as function approximators, show that all three parameters influence overestimation significantly. By carefully tuning $\alpha$ and $\gamma$, and by using an exponential moving average of $r$ in Q-Learning's temporal difference target, we show that the algorithm can learn value estimates that are more accurate than the ones of several other popular model-free methods that have addressed its overestimation bias in the past.