MVIP-NeRF: Multi-view 3D Inpainting on NeRF Scenes via Diffusion Prior

TL;DR

MVIP-NeRF introduces a diffusion prior-based multi-view inpainting method for NeRF scenes, achieving consistent appearance and geometry completion by jointly optimizing across views with iterative score distillation sampling.

Contribution

It presents a novel multi-view inpainting framework using diffusion priors and score distillation sampling to improve consistency and geometry accuracy in NeRF scene inpainting.

Findings

Outperforms previous NeRF inpainting methods in appearance recovery.

Achieves superior geometry completion and alignment.

Ensures consistent multi-view inpainting across large view variations.

Abstract

Despite the emergence of successful NeRF inpainting methods built upon explicit RGB and depth 2D inpainting supervisions, these methods are inherently constrained by the capabilities of their underlying 2D inpainters. This is due to two key reasons: (i) independently inpainting constituent images results in view-inconsistent imagery, and (ii) 2D inpainters struggle to ensure high-quality geometry completion and alignment with inpainted RGB images. To overcome these limitations, we propose a novel approach called MVIP-NeRF that harnesses the potential of diffusion priors for NeRF inpainting, addressing both appearance and geometry aspects. MVIP-NeRF performs joint inpainting across multiple views to reach a consistent solution, which is achieved via an iterative optimization process based on Score Distillation Sampling (SDS). Apart from recovering the rendered RGB images, we also extract normal maps as a geometric representation and define a normal SDS loss that motivates accurate geometry inpainting and alignment with the appearance. Additionally, we formulate a multi-view SDS score function to distill generative priors simultaneously from different view images, ensuring consistent visual completion when dealing with large view variations. Our experimental results show better appearance and geometry recovery than previous NeRF inpainting methods.