SceneCompleter: Dense 3D Scene Completion for Generative Novel View Synthesis

TL;DR

SceneCompleter introduces a 3D-consistent generative framework for novel view synthesis by dense scene completion, combining a dual-stream diffusion model with a holistic scene embedder for improved coherence and realism.

Contribution

It proposes a novel 3D scene completion framework that jointly synthesizes RGBD views and encodes holistic scene understanding, surpassing traditional 2D completion methods.

Findings

Achieves superior visual coherence in novel view synthesis.

Demonstrates improved 3D consistency and scene plausibility.

Outperforms existing methods across multiple datasets.

Abstract

Generative models have gained significant attention in novel view synthesis (NVS) by alleviating the reliance on dense multi-view captures. However, existing methods typically fall into a conventional paradigm, where generative models first complete missing areas in 2D, followed by 3D recovery techniques to reconstruct the scene, which often results in overly smooth surfaces and distorted geometry, as generative models struggle to infer 3D structure solely from RGB data. In this paper, we propose SceneCompleter, a novel framework that achieves 3D-consistent generative novel view synthesis through dense 3D scene completion. SceneCompleter achieves both visual coherence and 3D-consistent generative scene completion through two key components: (1) a geometry-appearance dual-stream diffusion model that jointly synthesizes novel views in RGBD space; (2) a scene embedder that encodes a more holistic scene understanding from the reference image. By effectively fusing structural and textural information, our method demonstrates superior coherence and plausibility in generative novel view synthesis across diverse datasets. Project Page: https://chen-wl20.github.io/SceneCompleter