Cross Learning in Deep Q-Networks

TL;DR

This paper introduces a cross Q-learning algorithm for deep Q-networks that reduces overestimation bias and stabilizes training by maintaining multiple models and using random network selection.

Contribution

The paper proposes a novel cross Q-learning method that extends double Q-learning with multiple models to mitigate overestimation in deep reinforcement learning.

Findings

Significant reduction in overestimation bias.

Improved training stability and performance.

Better policy quality on benchmark environments.

Abstract

In this work, we propose a novel cross Q-learning algorithm, aim at alleviating the well-known overestimation problem in value-based reinforcement learning methods, particularly in the deep Q-networks where the overestimation is exaggerated by function approximation errors. Our algorithm builds on double Q-learning, by maintaining a set of parallel models and estimate the Q-value based on a randomly selected network, which leads to reduced overestimation bias as well as the variance. We provide empirical evidence on the advantages of our method by evaluating on some benchmark environment, the experimental results demonstrate significant improvement of performance in reducing the overestimation bias and stabilizing the training, further leading to better derived policies.