Variance Reduction for Deep Q-Learning using Stochastic Recursive Gradient

TL;DR

This paper introduces SRG-DQN, a recursive variance reduction method for deep Q-learning that improves gradient estimation stability and training efficiency by eliminating the need for anchor points, supported by theoretical and experimental validation.

Contribution

The paper proposes a novel recursive gradient update algorithm for deep Q-learning, overcoming limitations of SVRG-based methods and enhancing training stability and efficiency.

Findings

SRG-DQN reduces gradient variance effectively.

The method accelerates training convergence.

Experimental results outperform existing algorithms.

Abstract

Deep Q-learning algorithms often suffer from poor gradient estimations with an excessive variance, resulting in unstable training and poor sampling efficiency. Stochastic variance-reduced gradient methods such as SVRG have been applied to reduce the estimation variance (Zhao et al. 2019). However, due to the online instance generation nature of reinforcement learning, directly applying SVRG to deep Q-learning is facing the problem of the inaccurate estimation of the anchor points, which dramatically limits the potentials of SVRG. To address this issue and inspired by the recursive gradient variance reduction algorithm SARAH (Nguyen et al. 2017), this paper proposes to introduce the recursive framework for updating the stochastic gradient estimates in deep Q-learning, achieving a novel algorithm called SRG-DQN. Unlike the SVRG-based algorithms, SRG-DQN designs a recursive update of the stochastic gradient estimate. The parameter update is along an accumulated direction using the past stochastic gradient information, and therefore can get rid of the estimation of the full gradients as the anchors. Additionally, SRG-DQN involves the Adam process for further accelerating the training process. Theoretical analysis and the experimental results on well-known reinforcement learning tasks demonstrate the efficiency and effectiveness of the proposed SRG-DQN algorithm.