On the Evaluation of Procedural Level Generation Systems

TL;DR

This paper analyzes current methods for evaluating procedural level generation in video games, proposing a taxonomy and highlighting weaknesses to improve consistency and robustness in future research.

Contribution

It introduces a novel taxonomy for PCG evaluation approaches and provides a survey of recent work to identify key weaknesses and suggest improvements.

Findings

Current evaluation practices are inconsistent and limited.

Promoting evaluation-free descriptions can improve comparability.

Encouraging code reuse and diverse frameworks enhances research robustness.

Abstract

The evaluation of procedural content generation (PCG) systems for generating video game levels is a complex and contested topic. Ideally, the field would have access to robust, generalisable and widely accepted evaluation approaches that can be used to compare novel PCG systems to prior work, but consensus on how to evaluate novel systems is currently limited. We argue that the field can benefit from a structured analysis of how procedural level generation systems can be evaluated, and how these techniques are currently used by researchers. This analysis can then be used to both inform on the current state of affairs, and to provide data to justify changes to this practice. This work aims to provide this by first developing a novel taxonomy of PCG evaluation approaches, and then presenting the results of a survey of recent work in the field through the lens of this taxonomy. The results of this survey highlight several important weaknesses in current practice which we argue could be substantially mitigated by 1) promoting use of evaluation free system descriptions where appropriate, 2) promoting the development of diverse research frameworks, 3) promoting reuse of code and methodology wherever possible.