Text-to-Level Diffusion Models With Various Text Encoders for Super Mario Bros

TL;DR

This paper explores text-to-level generation for Super Mario Bros using diffusion models with various text encoders, demonstrating that simple transformers can be effective and more efficient than complex models, with practical tools for level design.

Contribution

It introduces strategies for automatic captioning, compares different diffusion models and encoders, and provides a GUI for level construction, advancing text-to-level generation methods.

Findings

Simple transformer encoders outperform complex models in training time.

Diffusion models generate diverse and playable levels.

The proposed GUI aids designers in constructing long levels.

Abstract

Recent research shows how diffusion models can unconditionally generate tile-based game levels, but use of diffusion models for text-to-level generation is underexplored. There are practical considerations for creating a usable model: caption/level pairs are needed, as is a text embedding model, and a way of generating entire playable levels, rather than individual scenes. We present strategies to automatically assign descriptive captions to an existing dataset, and train diffusion models using both pretrained text encoders and simple transformer models trained from scratch. Captions are automatically assigned to generated scenes so that the degree of overlap between input and output captions can be compared. We also assess the diversity and playability of the resulting level scenes. Results are compared with an unconditional diffusion model and a generative adversarial network, as well as the text-to-level approaches Five-Dollar Model and MarioGPT. Notably, the best diffusion model uses a simple transformer model for text embedding, and takes less time to train than diffusion models employing more complex text encoders, indicating that reliance on larger language models is not necessary. We also present a GUI allowing designers to construct long levels from model-generated scenes.