Where and How: Mitigating Confusion in Neural Radiance Fields from Sparse Inputs

TL;DR

This paper introduces WaH-NeRF, a novel framework that reduces confusion in neural radiance fields from sparse inputs by addressing sampling and prediction challenges through new strategies and loss functions, improving view synthesis quality.

Contribution

WaH-NeRF presents a new learning framework with deformable sampling and semi-supervised training to mitigate confusion in sparse-input NeRFs, advancing view synthesis accuracy.

Findings

Outperforms previous methods in sparse-input NeRF tasks

Effectively reduces over-fitting and foggy surfaces

Improves novel view synthesis quality

Abstract

Neural Radiance Fields from Sparse input} (NeRF-S) have shown great potential in synthesizing novel views with a limited number of observed viewpoints. However, due to the inherent limitations of sparse inputs and the gap between non-adjacent views, rendering results often suffer from over-fitting and foggy surfaces, a phenomenon we refer to as "CONFUSION" during volume rendering. In this paper, we analyze the root cause of this confusion and attribute it to two fundamental questions: "WHERE" and "HOW". To this end, we present a novel learning framework, WaH-NeRF, which effectively mitigates confusion by tackling the following challenges: (i)"WHERE" to Sample? in NeRF-S -- we introduce a Deformable Sampling strategy and a Weight-based Mutual Information Loss to address sample-position confusion arising from the limited number of viewpoints; and (ii) "HOW" to Predict? in NeRF-S -- we propose a Semi-Supervised NeRF learning Paradigm based on pose perturbation and a Pixel-Patch Correspondence Loss to alleviate prediction confusion caused by the disparity between training and testing viewpoints. By integrating our proposed modules and loss functions, WaH-NeRF outperforms previous methods under the NeRF-S setting. Code is available https://github.com/bbbbby-99/WaH-NeRF.