DreamCraft: Text-Guided Generation of Functional 3D Environments in Minecraft
Sam Earle, Filippos Kokkinos, Yuhe Nie, Julian Togelius, Roberta, Raileanu

TL;DR
DreamCraft is a novel method that generates functional 3D Minecraft environments from natural language prompts by training quantized NeRFs, enabling high control and adherence to functional constraints.
Contribution
It introduces a neural radiance field-based approach for text-guided, functional 3D environment generation in Minecraft, combining expressivity with functional constraint integration.
Findings
Produces more aligned in-game artifacts than baseline methods.
Effectively incorporates functional constraints via specialized loss terms.
Generates environments matching target distributions and adjacency rules.
Abstract
Procedural Content Generation (PCG) algorithms enable the automatic generation of complex and diverse artifacts. However, they don't provide high-level control over the generated content and typically require domain expertise. In contrast, text-to-3D methods allow users to specify desired characteristics in natural language, offering a high amount of flexibility and expressivity. But unlike PCG, such approaches cannot guarantee functionality, which is crucial for certain applications like game design. In this paper, we present a method for generating functional 3D artifacts from free-form text prompts in the open-world game Minecraft. Our method, DreamCraft, trains quantized Neural Radiance Fields (NeRFs) to represent artifacts that, when viewed in-game, match given text descriptions. We find that DreamCraft produces more aligned in-game artifacts than a baseline that post-processes the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsHuman Motion and Animation · Image Processing and 3D Reconstruction · Augmented Reality Applications
