Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning

TL;DR

Averaged-DQN enhances deep reinforcement learning stability and performance by averaging past Q-value estimates, reducing variance and approximation errors, as demonstrated through theoretical analysis and Arcade Learning Environment experiments.

Contribution

The paper introduces Averaged-DQN, a simple yet effective extension to DQN that stabilizes training and improves performance by variance reduction.

Findings

Significantly improved stability in training.

Enhanced performance on Arcade Learning Environment.

Theoretical analysis supports variance reduction benefits.

Abstract

Instability and variability of Deep Reinforcement Learning (DRL) algorithms tend to adversely affect their performance. Averaged-DQN is a simple extension to the DQN algorithm, based on averaging previously learned Q-values estimates, which leads to a more stable training procedure and improved performance by reducing approximation error variance in the target values. To understand the effect of the algorithm, we examine the source of value function estimation errors and provide an analytical comparison within a simplified model. We further present experiments on the Arcade Learning Environment benchmark that demonstrate significantly improved stability and performance due to the proposed extension.