Joint Level Generation and Translation Using Gameplay Videos

TL;DR

This paper introduces a multi-task framework that simultaneously translates gameplay videos into level representations and generates new level segments, addressing data scarcity in procedural content generation.

Contribution

It presents a novel multi-tail framework that jointly learns level translation and generation from gameplay videos, reducing the need for laborious secondary representations.

Findings

Joint framework improves both translation and generation performance.

Combining tasks enhances overall results compared to baselines.

Potential to generalize to unseen games in future work.

Abstract

Procedural Content Generation via Machine Learning (PCGML) faces a significant hurdle that sets it apart from other fields, such as image or text generation, which is limited annotated data. Many existing methods for procedural level generation via machine learning require a secondary representation besides level images. However, the current methods for obtaining such representations are laborious and time-consuming, which contributes to this problem. In this work, we aim to address this problem by utilizing gameplay videos of two human-annotated games to develop a novel multi-tail framework that learns to perform simultaneous level translation and generation. The translation tail of our framework can convert gameplay video frames to an equivalent secondary representation, while its generation tail can produce novel level segments. Evaluation results and comparisons between our framework and baselines suggest that combining the level generation and translation tasks can lead to an overall improved performance regarding both tasks. This represents a possible solution to limited annotated level data, and we demonstrate the potential for future versions to generalize to unseen games.