State Space Closure: Revisiting Endless Online Level Generation via Reinforcement Learning

TL;DR

This paper analyzes how reinforcement learning-based procedural content generation can produce high-quality game levels with limited diversity due to state space closure, and provides theoretical and empirical insights on this phenomenon.

Contribution

It introduces the concept of state space closure, demonstrating that finite-horizon trained EDRL can be extended to infinite horizons without quality loss, but with limited diversity.

Findings

EDRL levels maintain quality over longer horizons.

Diversity of generated levels is limited by state space closure.

Theoretical analysis supports empirical observations.

Abstract

In this paper, we revisit endless online level generation with the recently proposed experience-driven procedural content generation via reinforcement learning (EDRL) framework. Inspired by an observation that EDRL tends to generate recurrent patterns, we formulate a notion of state space closure which makes any stochastic state appeared possibly in an infinite-horizon online generation process can be found within a finite-horizon. Through theoretical analysis, we find that even though state space closure arises a concern about diversity, it generalises EDRL trained with a finite-horizon to the infinite-horizon scenario without deterioration of content quality. Moreover, we verify the quality and the diversity of contents generated by EDRL via empirical studies, on the widely used Super Mario Bros. benchmark. Experimental results reveal that the diversity of levels generated by EDRL is limited due to the state space closure, whereas their quality does not deteriorate in a horizon which is longer than the one specified in the training. Concluding our outcomes and analysis, future work on endless online level generation via reinforcement learning should address the issue of diversity while assuring the occurrence of state space closure and quality.