PCGRL+: Scaling, Control and Generalization in Reinforcement Learning Level Generators

TL;DR

This paper enhances PCGRL by implementing GPU-accelerated environments in Jax, enabling faster training, larger scale level generation, and improved generalization, thus making PCGRL more practical for complex game design tasks.

Contribution

It introduces a GPU-based Jax implementation of PCGRL environments, allowing scalable, faster training and new control mechanisms for level generation.

Findings

Faster training speeds due to GPU parallelization.

Models trained for over 1 billion timesteps show robust behavior.

Partial observation strategies improve generalization to larger levels.

Abstract

Procedural Content Generation via Reinforcement Learning (PCGRL) has been introduced as a means by which controllable designer agents can be trained based only on a set of computable metrics acting as a proxy for the level's quality and key characteristics. While PCGRL offers a unique set of affordances for game designers, it is constrained by the compute-intensive process of training RL agents, and has so far been limited to generating relatively small levels. To address this issue of scale, we implement several PCGRL environments in Jax so that all aspects of learning and simulation happen in parallel on the GPU, resulting in faster environment simulation; removing the CPU-GPU transfer of information bottleneck during RL training; and ultimately resulting in significantly improved training speed. We replicate several key results from prior works in this new framework, letting models train for much longer than previously studied, and evaluating their behavior after 1 billion timesteps. Aiming for greater control for human designers, we introduce randomized level sizes and frozen "pinpoints" of pivotal game tiles as further ways of countering overfitting. To test the generalization ability of learned generators, we evaluate models on large, out-of-distribution map sizes, and find that partial observation sizes learn more robust design strategies.