Layout2Scene: 3D Semantic Layout Guided Scene Generation via Geometry and Appearance Diffusion Priors

TL;DR

Layout2Scene introduces a novel method for 3D scene generation guided by semantic layouts, leveraging diffusion priors for geometry and appearance to produce realistic, editable scenes with precise object placement.

Contribution

The paper proposes a two-stage scene generation framework using semantic layouts and diffusion models, enabling fine-grained control and realistic 3D scene synthesis.

Findings

Outperforms state-of-the-art methods in scene plausibility and realism

Enables flexible editing of generated scenes

Effectively leverages 2D diffusion priors for 3D scene generation

Abstract

3D scene generation conditioned on text prompts has significantly progressed due to the development of 2D diffusion generation models. However, the textual description of 3D scenes is inherently inaccurate and lacks fine-grained control during training, leading to implausible scene generation. As an intuitive and feasible solution, the 3D layout allows for precise specification of object locations within the scene. To this end, we present a text-to-scene generation method (namely, Layout2Scene) using additional semantic layout as the prompt to inject precise control of 3D object positions. Specifically, we first introduce a scene hybrid representation to decouple objects and backgrounds, which is initialized via a pre-trained text-to-3D model. Then, we propose a two-stage scheme to optimize the geometry and appearance of the initialized scene separately. To fully leverage 2D diffusion priors in geometry and appearance generation, we introduce a semantic-guided geometry diffusion model and a semantic-geometry guided diffusion model which are finetuned on a scene dataset. Extensive experiments demonstrate that our method can generate more plausible and realistic scenes as compared to state-of-the-art approaches. Furthermore, the generated scene allows for flexible yet precise editing, thereby facilitating multiple downstream applications.