An Empirical Comparison of Neural Architectures for Reinforcement Learning in Partially Observable Environments

TL;DR

This paper empirically compares different neural network architectures for reinforcement learning in partially observable environments, finding that GRU outperforms LSTM and MUT1, and that advantage learning improves results.

Contribution

It provides a comparative analysis of RNN architectures for reinforcement learning, highlighting the superior performance of GRU and the benefits of advantage learning.

Findings

GRU outperforms LSTM and MUT1 in most tasks.

Advantage learning yields better policies.

GRU requires fewer training episodes and less CPU time.

Abstract

This paper explores the performance of fitted neural Q iteration for reinforcement learning in several partially observable environments, using three recurrent neural network architectures: Long Short-Term Memory, Gated Recurrent Unit and MUT1, a recurrent neural architecture evolved from a pool of several thousands candidate architectures. A variant of fitted Q iteration, based on Advantage values instead of Q values, is also explored. The results show that GRU performs significantly better than LSTM and MUT1 for most of the problems considered, requiring less training episodes and less CPU time before learning a very good policy. Advantage learning also tends to produce better results.