MutualNeRF: Improve the Performance of NeRF under Limited Samples with Mutual Information Theory

TL;DR

MutualNeRF leverages Mutual Information Theory to enhance Neural Radiance Field performance with limited samples by strategically selecting viewpoints and maximizing information transfer, leading to improved 3D scene synthesis.

Contribution

This work introduces a theoretically grounded framework using Mutual Information to improve NeRF with limited data, including a greedy algorithm for viewpoint selection and regularization for few-shot synthesis.

Findings

Consistent improvement over state-of-the-art methods in limited sample scenarios

Effective viewpoint selection reduces mutual information overlap

Enhanced 3D scene synthesis quality with fewer samples

Abstract

This paper introduces MutualNeRF, a framework enhancing Neural Radiance Field (NeRF) performance under limited samples using Mutual Information Theory. While NeRF excels in 3D scene synthesis, challenges arise with limited data and existing methods that aim to introduce prior knowledge lack theoretical support in a unified framework. We introduce a simple but theoretically robust concept, Mutual Information, as a metric to uniformly measure the correlation between images, considering both macro (semantic) and micro (pixel) levels. For sparse view sampling, we strategically select additional viewpoints containing more non-overlapping scene information by minimizing mutual information without knowing ground truth images beforehand. Our framework employs a greedy algorithm, offering a near-optimal solution. For few-shot view synthesis, we maximize the mutual information between inferred images and ground truth, expecting inferred images to gain more relevant information from known images. This is achieved by incorporating efficient, plug-and-play regularization terms. Experiments under limited samples show consistent improvement over state-of-the-art baselines in different settings, affirming the efficacy of our framework.