Investigating Recurrence and Eligibility Traces in Deep Q-Networks

TL;DR

This paper explores how eligibility traces combined with recurrent neural networks can improve training efficiency in reinforcement learning for Atari games, emphasizing the role of optimization methods.

Contribution

It demonstrates the benefits of integrating eligibility traces with recurrent networks in Atari, highlighting their impact on training speed and performance.

Findings

Eligibility traces speed up training in some Atari games.

Recurrent networks combined with eligibility traces improve learning efficiency.

Optimization methods significantly influence training outcomes.

Abstract

Eligibility traces in reinforcement learning are used as a bias-variance trade-off and can often speed up training time by propagating knowledge back over time-steps in a single update. We investigate the use of eligibility traces in combination with recurrent networks in the Atari domain. We illustrate the benefits of both recurrent nets and eligibility traces in some Atari games, and highlight also the importance of the optimization used in the training.