Natural Gradient Deep Q-learning

TL;DR

This paper introduces NGDQN, a natural-gradient-based deep Q-learning algorithm that stabilizes training, reduces hyperparameter sensitivity, and outperforms or matches traditional DQN methods across control tasks.

Contribution

It presents a novel natural-gradient approach for deep Q-learning, demonstrating improved stability and reduced hyperparameter sensitivity without using target networks.

Findings

NGDQN outperforms DQN without target networks.

NGDQN matches DQN with target networks in performance.

NGDQN is less sensitive to hyperparameter tuning.

Abstract

We present a novel algorithm to train a deep Q-learning agent using natural-gradient techniques. We compare the original deep Q-network (DQN) algorithm to its natural-gradient counterpart, which we refer to as NGDQN, on a collection of classic control domains. Without employing target networks, NGDQN significantly outperforms DQN without target networks, and performs no worse than DQN with target networks, suggesting that NGDQN stabilizes training and can help reduce the need for additional hyperparameter tuning. We also find that NGDQN is less sensitive to hyperparameter optimization relative to DQN. Together these results suggest that natural-gradient techniques can improve value-function optimization in deep reinforcement learning.