Full Gradient DQN Reinforcement Learning: A Provably Convergent Scheme

TL;DR

This paper introduces Full Gradient DQN, a theoretically sound modification of the original DQN algorithm, demonstrating improved performance through analysis and empirical comparison.

Contribution

The paper proposes a new variant called Full Gradient DQN with a solid theoretical foundation, addressing issues in the original DQN.

Findings

FG-DQN shows better performance than DQN on sample problems

Theoretical analysis confirms convergence properties of FG-DQN

Comparison highlights advantages of the proposed scheme

Abstract

We analyze the DQN reinforcement learning algorithm as a stochastic approximation scheme using the o.d.e. (for 'ordinary differential equation') approach and point out certain theoretical issues. We then propose a modified scheme called Full Gradient DQN (FG-DQN, for short) that has a sound theoretical basis and compare it with the original scheme on sample problems. We observe a better performance for FG-DQN.