DreamGarden: A Designer Assistant for Growing Games from a Single Prompt

TL;DR

DreamGarden is an AI-powered system that transforms high-level user prompts into detailed game environments in Unreal Engine, enabling intuitive human-AI collaboration in game design.

Contribution

It introduces a hierarchical planning system driven by an LLM that decomposes prompts into actionable plans, integrating user interaction for creative game development.

Findings

Effective hierarchical planning from high-level prompts

User study insights on human-AI collaboration in game design

Potential for semi-autonomous game environment generation

Abstract

Coding assistants are increasingly leveraged in game design, both generating code and making high-level plans. To what degree can these tools align with developer workflows, and what new modes of human-computer interaction can emerge from their use? We present DreamGarden, an AI system capable of assisting with the development of diverse game environments in Unreal Engine. At the core of our method is an LLM-driven planner, capable of breaking down a single, high-level prompt -- a dream, memory, or imagined scenario provided by a human user -- into a hierarchical action plan, which is then distributed across specialized submodules facilitating concrete implementation. This system is presented to the user as a garden of plans and actions, both growing independently and responding to user intervention via seed prompts, pruning, and feedback. Through a user study, we explore design implications of this system, charting courses for future work in semi-autonomous assistants and open-ended simulation design.