CompNVS: Novel View Synthesis with Scene Completion

TL;DR

CompNVS presents a scalable neural framework for novel view synthesis that effectively completes incomplete 3D scenes from RGB-D images, achieving photorealistic results through scene understanding and inpainting.

Contribution

It introduces a novel 3D scene completion pipeline combining geometry and texture inpainting for improved novel view synthesis from incomplete RGB-D data.

Findings

Outperforms state-of-the-art methods in scene completion quality.

Produces photorealistic novel views with high consistency.

Effectively handles largely incomplete scene coverage.

Abstract

We introduce a scalable framework for novel view synthesis from RGB-D images with largely incomplete scene coverage. While generative neural approaches have demonstrated spectacular results on 2D images, they have not yet achieved similar photorealistic results in combination with scene completion where a spatial 3D scene understanding is essential. To this end, we propose a generative pipeline performing on a sparse grid-based neural scene representation to complete unobserved scene parts via a learned distribution of scenes in a 2.5D-3D-2.5D manner. We process encoded image features in 3D space with a geometry completion network and a subsequent texture inpainting network to extrapolate the missing area. Photorealistic image sequences can be finally obtained via consistency-relevant differentiable rendering. Comprehensive experiments show that the graphical outputs of our method outperform the state of the art, especially within unobserved scene parts.