Leveraging Procedural Generation to Benchmark Reinforcement Learning

TL;DR

This paper presents the Procgen Benchmark, a set of procedurally generated environments for evaluating reinforcement learning agents on sample efficiency and generalization, highlighting the importance of environment diversity and model scaling.

Contribution

The paper introduces a new benchmark suite for RL, provides experimental protocols, and demonstrates the benefits of procedural generation and larger models for RL performance.

Findings

Diverse environment distributions are crucial for effective RL training and evaluation.

Procedural content generation enhances the robustness of RL benchmarks.

Scaling model size improves both sample efficiency and generalization in RL agents.

Abstract

We introduce Procgen Benchmark, a suite of 16 procedurally generated game-like environments designed to benchmark both sample efficiency and generalization in reinforcement learning. We believe that the community will benefit from increased access to high quality training environments, and we provide detailed experimental protocols for using this benchmark. We empirically demonstrate that diverse environment distributions are essential to adequately train and evaluate RL agents, thereby motivating the extensive use of procedural content generation. We then use this benchmark to investigate the effects of scaling model size, finding that larger models significantly improve both sample efficiency and generalization.