RGBD2: Generative Scene Synthesis via Incremental View Inpainting using RGBD Diffusion Models

TL;DR

RGBD2 introduces a novel method for 3D scene synthesis from sparse RGBD views by sequentially generating and inpainting views using a modified RGBD diffusion model, effectively addressing multi-view inconsistency.

Contribution

The paper presents RGBD2, a new approach that combines incremental view synthesis, mesh-based inpainting, and a modified diffusion model for improved 3D scene reconstruction from sparse data.

Findings

Outperforms existing methods on ScanNet dataset

Effectively handles multi-view inconsistency

Demonstrates high-quality 3D scene synthesis from sparse inputs

Abstract

We address the challenge of recovering an underlying scene geometry and colors from a sparse set of RGBD view observations. In this work, we present a new solution termed RGBD$^2$ that sequentially generates novel RGBD views along a camera trajectory, and the scene geometry is simply the fusion result of these views. More specifically, we maintain an intermediate surface mesh used for rendering new RGBD views, which subsequently becomes complete by an inpainting network; each rendered RGBD view is later back-projected as a partial surface and is supplemented into the intermediate mesh. The use of intermediate mesh and camera projection helps solve the tough problem of multi-view inconsistency. We practically implement the RGBD inpainting network as a versatile RGBD diffusion model, which is previously used for 2D generative modeling; we make a modification to its reverse diffusion process to enable our use. We evaluate our approach on the task of 3D scene synthesis from sparse RGBD inputs; extensive experiments on the ScanNet dataset demonstrate the superiority of our approach over existing ones. Project page: https://jblei.site/proj/rgbd-diffusion.