Learning Local Constraints for Reinforcement-Learned Content Generators

TL;DR

This paper presents a hybrid content generation approach combining local constraints learned by Wave Function Collapse with reinforcement learning to produce visually satisfying and playable game levels with guaranteed global properties.

Contribution

It introduces a method to constrain reinforcement learning generators with local constraints learned from existing content, improving visual quality and playability.

Findings

Best models produce satisfying, playable levels like Lode Runner.

Hybrid approach guarantees global properties while maintaining visual quality.

Method is sensitive to hyperparameter tuning.

Abstract

Constraint-based game content generators that learn local constraints from existing content, such as Wave Function Collapse (WFC), can generate visually satisfying game levels but face challenges in guaranteeing global properties, such as playability. On the other hand, reinforcement-learning trained generators can guarantee global properties -- because such properties can easily be included in reward functions -- but the results can be visually dissatisfying. In this paper, we explore ways to combine these methods. Specifically, we constrain the action space of a PCGRL generator with constraints learned by WFC, effectively allowing the PCGRL generator to achieve global properties while forced to adhere to local constraints. To better analyze how this hybrid content generation method operates, we vary the number and type of inputs, and we test whether to randomly collapse the starting state and exclude rare patterns. While the method is sensitive to hyperparameter tuning, the best of our trained generators produce visually satisfying and playable puzzle-platform game levels -- such as Lode Runner levels -- with desired global properties.