Does DQN Learn?

TL;DR

This paper demonstrates that Deep Q-Networks (DQN) can fail to improve upon initial policies even with unlimited data, and provides a theoretical explanation for this sub-optimality using linear DQN analysis.

Contribution

It shows empirically that DQN can perform worse than the initial policy and offers a theoretical framework explaining why linear DQN converges to sub-optimal fixed points.

Findings

DQN often yields policies worse than initial guesses.

Linear DQN's limit points are fixed points of projected Bellman operators.

Fixed points may not be near-optimal or even good policies.

Abstract

A primary requirement for any reinforcement learning method is that it should produce policies that improve upon the initial guess. In this work, we show that the widely used Deep Q-Network (DQN) fails to satisfy this minimal criterion -- even when it gets to see all possible states and actions infinitely often (a condition under which tabular Q-learning is guaranteed to converge to the optimal Q-value function). Our specific contributions are twofold. First, we numerically show that DQN often returns a policy that performs worse than the initial one. Second, we offer a theoretical explanation for this phenomenon in linear DQN, a simplified version of DQN that uses linear function approximation in place of neural networks while retaining the other key components such as $\epsilon$-greedy exploration, experience replay, and target network. Using tools from differential inclusion theory, we prove that the limit points of linear DQN correspond to fixed points of projected Bellman operators. Crucially, we show that these fixed points need not relate to optimal -- or even near-optimal -- policies, thus explaining linear DQN's sub-optimal behaviors. We also give a scenario where linear DQN always identifies the worst policy. Our work fills a longstanding gap in understanding the convergence behaviors of Q-learning with function approximation and $\epsilon$-greedy exploration.