RoomDreamer: Text-Driven 3D Indoor Scene Synthesis with Coherent Geometry and Texture

TL;DR

RoomDreamer is a novel framework that synthesizes 3D indoor scenes from text prompts, ensuring coherent geometry and texture by integrating geometry guidance and mesh optimization, validated on real scanned scenes.

Contribution

The paper introduces a new text-driven 3D scene synthesis method that jointly optimizes geometry and texture for realistic indoor environments.

Findings

Effective synthesis of 3D indoor scenes with consistent style.

Improved geometry and texture quality in scanned scenes.

Validated on real smartphone-scanned indoor environments.

Abstract

The techniques for 3D indoor scene capturing are widely used, but the meshes produced leave much to be desired. In this paper, we propose "RoomDreamer", which leverages powerful natural language to synthesize a new room with a different style. Unlike existing image synthesis methods, our work addresses the challenge of synthesizing both geometry and texture aligned to the input scene structure and prompt simultaneously. The key insight is that a scene should be treated as a whole, taking into account both scene texture and geometry. The proposed framework consists of two significant components: Geometry Guided Diffusion and Mesh Optimization. Geometry Guided Diffusion for 3D Scene guarantees the consistency of the scene style by applying the 2D prior to the entire scene simultaneously. Mesh Optimization improves the geometry and texture jointly and eliminates the artifacts in the scanned scene. To validate the proposed method, real indoor scenes scanned with smartphones are used for extensive experiments, through which the effectiveness of our method is demonstrated.