NeRF-MIR: Towards High-Quality Restoration of Masked Images with Neural Radiance Fields

TL;DR

NeRF-MIR introduces a novel neural rendering method utilizing patch-based entropy and progressive iterative restoration to effectively restore masked regions in images, demonstrating superior performance on constructed and real datasets.

Contribution

The paper proposes NeRF-MIR, a new approach combining PERE and PIRE strategies for high-quality masked image restoration using neural radiance fields.

Findings

NeRF-MIR outperforms existing methods in masked image restoration.

Constructed three new masked datasets for evaluation.

Demonstrates effectiveness on real and synthetic corrupted images.

Abstract

Neural Radiance Fields (NeRF) have demonstrated remarkable performance in novel view synthesis. However, there is much improvement room on restoring 3D scenes based on NeRF from corrupted images, which are common in natural scene captures and can significantly impact the effectiveness of NeRF. This paper introduces NeRF-MIR, a novel neural rendering approach specifically proposed for the restoration of masked images, demonstrating the potential of NeRF in this domain. Recognizing that randomly emitting rays to pixels in NeRF may not effectively learn intricate image textures, we propose a \textbf{P}atch-based \textbf{E}ntropy for \textbf{R}ay \textbf{E}mitting (\textbf{PERE}) strategy to distribute emitted rays properly. This enables NeRF-MIR to fuse comprehensive information from images of different views. Additionally, we introduce a \textbf{P}rogressively \textbf{I}terative \textbf{RE}storation (\textbf{PIRE}) mechanism to restore the masked regions in a self-training process. Furthermore, we design a dynamically-weighted loss function that automatically recalibrates the loss weights for masked regions. As existing datasets do not support NeRF-based masked image restoration, we construct three masked datasets to simulate corrupted scenarios. Extensive experiments on real data and constructed datasets demonstrate the superiority of NeRF-MIR over its counterparts in masked image restoration.