LAYOUTDREAMER: Physics-guided Layout for Text-to-3D Compositional Scene Generation

TL;DR

LayoutDreamer introduces a physics-guided, controllable framework for text-to-3D scene generation that improves realism, compositionality, and semantic alignment using 3D Gaussian Splatting and scene graph-based adjustments.

Contribution

The paper presents LayoutDreamer, a novel framework that enhances text-guided 3D scene generation by integrating physical constraints and scene graph-based layout adjustments.

Findings

Outperforms existing methods in scene quality and semantic alignment.

Achieves state-of-the-art results on T3Bench multiple objects metric.

Demonstrates high physical plausibility and controllability in generated scenes.

Abstract

Recently, the field of text-guided 3D scene generation has garnered significant attention. High-quality generation that aligns with physical realism and high controllability is crucial for practical 3D scene applications. However, existing methods face fundamental limitations: (i) difficulty capturing complex relationships between multiple objects described in the text, (ii) inability to generate physically plausible scene layouts, and (iii) lack of controllability and extensibility in compositional scenes. In this paper, we introduce LayoutDreamer, a framework that leverages 3D Gaussian Splatting (3DGS) to facilitate high-quality, physically consistent compositional scene generation guided by text. Specifically, given a text prompt, we convert it into a directed scene graph and adaptively adjust the density and layout of the initial compositional 3D Gaussians. Subsequently, dynamic camera adjustments are made based on the training focal point to ensure entity-level generation quality. Finally, by extracting directed dependencies from the scene graph, we tailor physical and layout energy to ensure both realism and flexibility. Comprehensive experiments demonstrate that LayoutDreamer outperforms other compositional scene generation quality and semantic alignment methods. Specifically, it achieves state-of-the-art (SOTA) performance in the multiple objects generation metric of T3Bench.