NeRF Inpainting with Geometric Diffusion Prior and Balanced Score Distillation

TL;DR

This paper introduces GB-NeRF, a novel framework that improves NeRF inpainting by better utilizing 2D diffusion priors through a fine-tuning strategy and a new balanced score distillation method, leading to enhanced appearance and geometric quality.

Contribution

The paper proposes a new GB-NeRF framework with a fine-tuning strategy and balanced score distillation, addressing limitations of previous diffusion-based NeRF inpainting methods.

Findings

Superior inpainting quality in appearance and geometry.

Outperforms existing methods like SDS and CSD.

Extensive experiments validate improved fidelity and consistency.

Abstract

Recent advances in NeRF inpainting have leveraged pretrained diffusion models to enhance performance. However, these methods often yield suboptimal results due to their ineffective utilization of 2D diffusion priors. The limitations manifest in two critical aspects: the inadequate capture of geometric information by pretrained diffusion models and the suboptimal guidance provided by existing Score Distillation Sampling (SDS) methods. To address these problems, we introduce GB-NeRF, a novel framework that enhances NeRF inpainting through improved utilization of 2D diffusion priors. Our approach incorporates two key innovations: a fine-tuning strategy that simultaneously learns appearance and geometric priors and a specialized normal distillation loss that integrates these geometric priors into NeRF inpainting. We propose a technique called Balanced Score Distillation (BSD) that surpasses existing methods such as Score Distillation (SDS) and the improved version, Conditional Score Distillation (CSD). BSD offers improved inpainting quality in appearance and geometric aspects. Extensive experiments show that our method provides superior appearance fidelity and geometric consistency compared to existing approaches.