MMGDreamer: Mixed-Modality Graph for Geometry-Controllable 3D Indoor Scene Generation

TL;DR

MMGDreamer introduces a dual-branch diffusion model with a mixed-modality graph to enable precise, flexible, and high-fidelity 3D indoor scene generation with controllable object geometry.

Contribution

It proposes a novel mixed-modality graph and visual enhancement modules for improved control and adaptability in 3D scene generation.

Findings

Achieves state-of-the-art scene generation performance.

Demonstrates superior control over object geometry.

Enables flexible user input integration.

Abstract

Controllable 3D scene generation has extensive applications in virtual reality and interior design, where the generated scenes should exhibit high levels of realism and controllability in terms of geometry. Scene graphs provide a suitable data representation that facilitates these applications. However, current graph-based methods for scene generation are constrained to text-based inputs and exhibit insufficient adaptability to flexible user inputs, hindering the ability to precisely control object geometry. To address this issue, we propose MMGDreamer, a dual-branch diffusion model for scene generation that incorporates a novel Mixed-Modality Graph, visual enhancement module, and relation predictor. The mixed-modality graph allows object nodes to integrate textual and visual modalities, with optional relationships between nodes. It enhances adaptability to flexible user inputs and enables meticulous control over the geometry of objects in the generated scenes. The visual enhancement module enriches the visual fidelity of text-only nodes by constructing visual representations using text embeddings. Furthermore, our relation predictor leverages node representations to infer absent relationships between nodes, resulting in more coherent scene layouts. Extensive experimental results demonstrate that MMGDreamer exhibits superior control of object geometry, achieving state-of-the-art scene generation performance. Project page: https://yangzhifeio.github.io/project/MMGDreamer.