Dueling Network Architectures for Deep Reinforcement Learning

TL;DR

This paper introduces a dueling network architecture for deep reinforcement learning that separately estimates state values and action advantages, improving policy evaluation and outperforming existing methods on Atari games.

Contribution

The paper proposes a novel neural network architecture that factors value and advantage estimations, enhancing learning efficiency without altering the underlying RL algorithm.

Findings

Improved policy evaluation with many similar-valued actions.

Outperforms state-of-the-art on Atari 2600 games.

Enables better generalization across actions.

Abstract

In recent years there have been many successes of using deep representations in reinforcement learning. Still, many of these applications use conventional architectures, such as convolutional networks, LSTMs, or auto-encoders. In this paper, we present a new neural network architecture for model-free reinforcement learning. Our dueling network represents two separate estimators: one for the state value function and one for the state-dependent action advantage function. The main benefit of this factoring is to generalize learning across actions without imposing any change to the underlying reinforcement learning algorithm. Our results show that this architecture leads to better policy evaluation in the presence of many similar-valued actions. Moreover, the dueling architecture enables our RL agent to outperform the state-of-the-art on the Atari 2600 domain.