ManifoldNeRF: View-dependent Image Feature Supervision for Few-shot Neural Radiance Fields

TL;DR

ManifoldNeRF improves few-shot neural radiance field synthesis by interpolating features from neighboring viewpoints, leading to better volume representation and performance in complex scenes.

Contribution

It introduces a novel supervision method using interpolated features to address viewpoint-dependent feature changes in NeRF training.

Findings

Outperforms DietNeRF in complex scene synthesis.

Effective viewpoint subset selection for real environments.

Provides a basic policy for viewpoint pattern selection.

Abstract

Novel view synthesis has recently made significant progress with the advent of Neural Radiance Fields (NeRF). DietNeRF is an extension of NeRF that aims to achieve this task from only a few images by introducing a new loss function for unknown viewpoints with no input images. The loss function assumes that a pre-trained feature extractor should output the same feature even if input images are captured at different viewpoints since the images contain the same object. However, while that assumption is ideal, in reality, it is known that as viewpoints continuously change, also feature vectors continuously change. Thus, the assumption can harm training. To avoid this harmful training, we propose ManifoldNeRF, a method for supervising feature vectors at unknown viewpoints using interpolated features from neighboring known viewpoints. Since the method provides appropriate supervision for each unknown viewpoint by the interpolated features, the volume representation is learned better than DietNeRF. Experimental results show that the proposed method performs better than others in a complex scene. We also experimented with several subsets of viewpoints from a set of viewpoints and identified an effective set of viewpoints for real environments. This provided a basic policy of viewpoint patterns for real-world application. The code is available at https://github.com/haganelego/ManifoldNeRF_BMVC2023