DecoRec: Decomposed 3D Scene Reconstruction from Single-View Images via Object-Level Diffusion

TL;DR

DecoRec is a novel system that reconstructs high-quality 3D scenes from single-view images by decomposing the scene into objects, reconstructing them individually, and merging them through a diffusion-guided refinement process.

Contribution

DecoRec introduces a new approach combining object-level diffusion reconstruction with a refinement pipeline for accurate 3D scene generation from single images.

Findings

Enables high-quality 3D scene reconstruction in geometry and appearance.

Improves scene synthesis for applications like interior design.

Outperforms existing methods in single-view scene reconstruction.

Abstract

In this paper, we introduce \textit{DecoRec}, a novel system designed to elevate single-view 2D images to a decomposed 3D scene mesh. Current methods for single-view scene reconstruction typically rely on object retrieval or the regression of coarse 3D voxels or surfaces, leading to inaccuracies in capturing the appearance and geometry of the input image. The lack of high-quality large-scale scene-level datasets further complicates direct 3D scene generation from single-view images. To achieve high-quality 3D scene generation from a single-view image, DecoRec takes advantage of recent diffusion-based single-view object reconstruction methods to reconstruct individual objects separately. Subsequently, a refinement pipeline is proposed to effectively merge these reconstructed objects, enhancing appearance and geometry through a differentiable rendering technique and diffusion-guided refinement. Our results demonstrate that DecoRec facilitates high-quality single-view scene reconstruction in both geometry and novel synthesis, offering significant benefits for downstream applications like room interior design.