Coherent 3D Scene Diffusion From a Single RGB Image

TL;DR

This paper introduces a diffusion-based method for reconstructing coherent 3D scenes from a single RGB image, capturing object relationships and scene context to improve accuracy over previous methods.

Contribution

The authors propose a novel scene diffusion model conditioned on all scene objects, with an efficient surface alignment loss that enables training without full ground-truth annotations.

Findings

Achieves 12.04% improvement in AP3D on SUN RGB-D

Achieves 13.43% increase in F-Score on Pix3D

Surpasses current state-of-the-art methods in 3D scene reconstruction

Abstract

We present a novel diffusion-based approach for coherent 3D scene reconstruction from a single RGB image. Our method utilizes an image-conditioned 3D scene diffusion model to simultaneously denoise the 3D poses and geometries of all objects within the scene. Motivated by the ill-posed nature of the task and to obtain consistent scene reconstruction results, we learn a generative scene prior by conditioning on all scene objects simultaneously to capture the scene context and by allowing the model to learn inter-object relationships throughout the diffusion process. We further propose an efficient surface alignment loss to facilitate training even in the absence of full ground-truth annotation, which is common in publicly available datasets. This loss leverages an expressive shape representation, which enables direct point sampling from intermediate shape predictions. By framing the task of single RGB image 3D scene reconstruction as a conditional diffusion process, our approach surpasses current state-of-the-art methods, achieving a 12.04% improvement in AP3D on SUN RGB-D and a 13.43% increase in F-Score on Pix3D.