UP-NeRF: Unconstrained Pose-Prior-Free Neural Radiance Fields

TL;DR

UP-NeRF introduces a method to optimize neural radiance fields from unconstrained image collections without relying on camera pose priors, effectively handling challenging real-world scenarios with varying illumination and occlusions.

Contribution

The paper proposes UP-NeRF, a novel approach that removes the need for pose priors in NeRF training, using surrogate tasks and modules for transient occluders and robust pose estimation.

Findings

Outperforms BARF and variants on Phototourism dataset

Effectively handles unconstrained images with occlusions and illumination changes

Demonstrates robustness in challenging real-world photo collections

Abstract

Neural Radiance Field (NeRF) has enabled novel view synthesis with high fidelity given images and camera poses. Subsequent works even succeeded in eliminating the necessity of pose priors by jointly optimizing NeRF and camera pose. However, these works are limited to relatively simple settings such as photometrically consistent and occluder-free image collections or a sequence of images from a video. So they have difficulty handling unconstrained images with varying illumination and transient occluders. In this paper, we propose $\textbf{UP-NeRF}$ ($\textbf{U}$nconstrained $\textbf{P}$ose-prior-free $\textbf{Ne}$ural $\textbf{R}$adiance $\textbf{F}$ields) to optimize NeRF with unconstrained image collections without camera pose prior. We tackle these challenges with surrogate tasks that optimize color-insensitive feature fields and a separate module for transient occluders to block their influence on pose estimation. In addition, we introduce a candidate head to enable more robust pose estimation and transient-aware depth supervision to minimize the effect of incorrect prior. Our experiments verify the superior performance of our method compared to the baselines including BARF and its variants in a challenging internet photo collection, $\textit{Phototourism}$ dataset.