HOLODECK 2.0: Vision-Language-Guided 3D World Generation with Editing

TL;DR

HOLODECK 2.0 is a vision-language-guided framework that generates and edits diverse, high-quality 3D scenes from detailed descriptions, improving automation and flexibility in 3D world creation.

Contribution

It introduces a novel interactive 3D scene generation and editing system leveraging vision-language models and state-of-the-art 3D generative models, supporting open-domain and style-rich environments.

Findings

Outperforms baselines in scene quality and semantic fidelity

Supports flexible scene editing based on human feedback

Effective in diverse styles and open-domain scenarios

Abstract

3D scene generation plays a crucial role in gaming, artistic creation, virtual reality, and many other domains. However, current 3D scene design still relies heavily on extensive manual effort from creators, and existing automated methods struggle to generate open-domain scenes or support flexible editing. To address those challenges, we introduce HOLODECK 2.0, an advanced vision-language-guided framework for 3D world generation with support for interactive scene editing based on human feedback. HOLODECK 2.0 can generate diverse and stylistically rich 3D scenes (e.g., realistic, cartoon, anime, and cyberpunk styles) that exhibit high semantic fidelity to fine-grained input descriptions, suitable for both indoor and open-domain environments. HOLODECK 2.0 leverages vision-language models (VLMs) to identify and parse the objects required in a scene and generates corresponding high-quality assets via state-of-the-art 3D generative models. Then, HOLODECK 2.0 iteratively applies spatial constraints derived from the VLMs to achieve semantically coherent and physically plausible layouts. Both human and model evaluations demonstrate that HOLODECK 2.0 effectively generates high-quality scenes closely aligned with detailed textual descriptions, consistently outperforming baselines across indoor and open-domain scenarios. Additionally, HOLODECK 2.0 provides editing capabilities that flexibly adapt to human feedback, supporting layout refinement and style-consistent object edits. Finally, we present a practical application of HOLODECK 2.0 in procedural game modeling to generate visually rich and immersive environments that can boost efficiency in game design.