3D Building Generation in Minecraft via Large Language Models

TL;DR

This paper demonstrates how large language models can be used to generate complete, structurally correct 3D buildings in Minecraft, supporting detailed features and meeting user specifications.

Contribution

It introduces the T2BM model that refines prompts and repairs generated structures, advancing LLM applications from 2D levels to 3D building generation in sandbox games.

Findings

LLMs can generate complete Minecraft buildings with correct structures.

Generated buildings meet user requirements and include specific features.

The approach shows significant potential for 3D content creation using LLMs.

Abstract

Recently, procedural content generation has exhibited considerable advancements in the domain of 2D game level generation such as Super Mario Bros. and Sokoban through large language models (LLMs). To further validate the capabilities of LLMs, this paper explores how LLMs contribute to the generation of 3D buildings in a sandbox game, Minecraft. We propose a Text to Building in Minecraft (T2BM) model, which involves refining prompts, decoding interlayer representation and repairing. Facade, indoor scene and functional blocks like doors are supported in the generation. Experiments are conducted to evaluate the completeness and satisfaction of buildings generated via LLMs. It shows that LLMs hold significant potential for 3D building generation. Given appropriate prompts, LLMs can generate correct buildings in Minecraft with complete structures and incorporate specific building blocks such as windows and beds, meeting the specified requirements of human users.