Higher Resolution, Better Generalization: Unlocking Visual Scaling in Deep Reinforcement Learning

TL;DR

Higher-resolution visual inputs significantly enhance deep reinforcement learning performance and generalization, especially with architectures that effectively process detailed visual information.

Contribution

This work demonstrates the importance of input resolution and proposes architecture modifications that decouple parameter growth from resolution, enabling better scaling.

Findings

Higher resolution inputs improve performance and generalization in deep RL.

Replacing flattening with global average pooling enables resolution scaling without parameter explosion.

Visual scaling yields a 28% performance increase over traditional architectures.

Abstract

Pixel-based deep reinforcement learning agents are typically trained on heavily downsampled visual observations, a convention inherited from early benchmarks rather than grounded in principled design. In this work, we show that observation resolution is a critical yet overlooked variable for policy learning: higher-resolution inputs can substantially improve both performance and generalization, provided the network architecture can process them effectively. We find that the widely used Impala encoder, which flattens spatial features into a vector, suffers from quadratic parameter growth as resolution increases and fails to leverage the additional visual detail. Replacing this operation with global average pooling, as in the Impoola architecture, decouples parameter count from resolution and yields consistent improvements across resolutions and network widths - at their respective best conditions, visual scaling unlocks a 28 % performance gain for Impoola over Impala. These gains are strongest in environments that require precise perception of small or distant objects, and gradient saliency analysis confirms that the underlying mechanism is a more spatially localized visual attention of the policy at higher resolutions. Our results challenge the prevailing practice of aggressive input downsampling and position resolution-independent architectures as a simple, effective path toward scalable visual deep RL. To facilitate future research on resolution scaling in deep RL, we publicly release the open-source code for the Procgen-HD benchmark: https://github.com/raphajaner/procgen-hd.