AutoUE: Automated Generation of 3D Games in Unreal Engine via Multi-Agent Systems

TL;DR

AutoUE is a multi-agent system that automates the end-to-end creation of 3D games in Unreal Engine, integrating asset generation, code synthesis, and testing.

Contribution

It introduces a novel multi-agent framework with retrieval-augmented generation and automated testing for systematic 3D game creation in Unreal Engine.

Findings

AutoUE successfully generates complete 3D games end-to-end.

The retrieval mechanism reduces hallucinations in code generation.

Automated testing evaluates dynamic game behaviors effectively.

Abstract

Automatically generating 3D games in commercial game engines remains a non-trivial challenge, as it involves complex engine-related workflows for generating assets such as scenes, blueprints, and code. To address this challenge, we propose a novel multi-agent system, AutoUE, which coordinates multiple agents to end-to-end generate 3D games, covering model retrieval, scene generation, gameplay and interaction code synthesis, and automated game testing for evaluation. In order to mitigate tool-use hallucinations in LLMs, we introduce a retrieval-augmented generation mechanism that grounds agents with relevant UE tool documentation. Additionally, we incorporate game design patterns and engine constraints into the code generation process to ensure the generation of correct and robust code. Furthermore, we design an automated play-testing pipeline that generates and executes runtime test commands, enabling systematic evaluation of dynamic behaviors. Finally, we construct a game generation dataset and conduct a series of experiments that demonstrate AutoUE's ability to generate 3D games end-to-end, and validate the effectiveness of these designs.