High-fidelity 3D Gaussian Inpainting: preserving multi-view consistency and photorealistic details

TL;DR

This paper introduces a novel 3D Gaussian inpainting framework that enhances multi-view consistency and photorealistic detail preservation in 3D scene reconstruction by leveraging sparse views, mask refinement, and uncertainty-guided optimization.

Contribution

It presents a new inpainting method combining automatic mask refinement and uncertainty-guided optimization to improve 3D scene completion and multi-view consistency.

Findings

Outperforms state-of-the-art methods in visual quality.

Achieves superior multi-view consistency.

Enhances fine detail fidelity in inpainted 3D scenes.

Abstract

Recent advancements in multi-view 3D reconstruction and novel-view synthesis, particularly through Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), have greatly enhanced the fidelity and efficiency of 3D content creation. However, inpainting 3D scenes remains a challenging task due to the inherent irregularity of 3D structures and the critical need for maintaining multi-view consistency. In this work, we propose a novel 3D Gaussian inpainting framework that reconstructs complete 3D scenes by leveraging sparse inpainted views. Our framework incorporates an automatic Mask Refinement Process and region-wise Uncertainty-guided Optimization. Specifically, we refine the inpainting mask using a series of operations, including Gaussian scene filtering and back-projection, enabling more accurate localization of occluded regions and realistic boundary restoration. Furthermore, our Uncertainty-guided Fine-grained Optimization strategy, which estimates the importance of each region across multi-view images during training, alleviates multi-view inconsistencies and enhances the fidelity of fine details in the inpainted results. Comprehensive experiments conducted on diverse datasets demonstrate that our approach outperforms existing state-of-the-art methods in both visual quality and view consistency.