DiffInDScene: Diffusion-based High-Quality 3D Indoor Scene Generation

TL;DR

DiffInDScene introduces an efficient diffusion-based framework for high-quality 3D indoor scene generation, capable of creating detailed room environments from scratch and refining multi-view stereo reconstructions.

Contribution

It presents a novel cascaded 3D diffusion pipeline that operates on sparse occupancy space and fuses local TSDF volumes for room-level 3D scene generation.

Findings

Achieves high-quality room generation directly in 3D space.

Outperforms ground truth meshes in user studies.

Provides a post-processing module for 3D reconstruction refinement.

Abstract

We present DiffInDScene, a novel framework for tackling the problem of high-quality 3D indoor scene generation, which is challenging due to the complexity and diversity of the indoor scene geometry. Although diffusion-based generative models have previously demonstrated impressive performance in image generation and object-level 3D generation, they have not yet been applied to room-level 3D generation due to their computationally intensive costs. In DiffInDScene, we propose a cascaded 3D diffusion pipeline that is efficient and possesses strong generative performance for Truncated Signed Distance Function (TSDF). The whole pipeline is designed to run on a sparse occupancy space in a coarse-to-fine fashion. Inspired by KinectFusion's incremental alignment and fusion of local TSDF volumes, we propose a diffusion-based SDF fusion approach that iteratively diffuses and fuses local TSDF volumes, facilitating the generation of an entire room environment. The generated results demonstrate that our work is capable to achieve high-quality room generation directly in three-dimensional space, starting from scratch. In addition to the scene generation, the final part of DiffInDScene can be used as a post-processing module to refine the 3D reconstruction results from multi-view stereo. According to the user study, the mesh quality generated by our DiffInDScene can even outperform the ground truth mesh provided by ScanNet. Please visit our project page for the latest progress and demonstrations: https://github.com/AkiraHero/diffindscene.