Mind the GAP! The Challenges of Scale in Pixel-based Deep Reinforcement Learning

TL;DR

This paper investigates the challenges of scaling pixel-based deep reinforcement learning, identifies the encoder-to-dense layer connection as a key bottleneck, and proposes global average pooling as a simple solution to improve scalability.

Contribution

It uncovers the encoder-dense layer connection as the main bottleneck in scaling, and introduces global average pooling as an effective, simple remedy.

Findings

Global average pooling improves scaling performance.

The encoder-to-dense layer connection is the primary bottleneck.

Previous methods implicitly targeted this bottleneck.

Abstract

Scaling deep reinforcement learning in pixel-based environments presents a significant challenge, often resulting in diminished performance. While recent works have proposed algorithmic and architectural approaches to address this, the underlying cause of the performance drop remains unclear. In this paper, we identify the connection between the output of the encoder (a stack of convolutional layers) and the ensuing dense layers as the main underlying factor limiting scaling capabilities; we denote this connection as the bottleneck, and we demonstrate that previous approaches implicitly target this bottleneck. As a result of our analyses, we present global average pooling as a simple yet effective way of targeting the bottleneck, thereby avoiding the complexity of earlier approaches.