DaRF: Boosting Radiance Fields from Sparse Inputs with Monocular Depth Adaptation

TL;DR

DaRF introduces a novel framework that combines monocular depth estimation with neural radiance fields, enabling high-quality 3D reconstruction from very few images by addressing depth ambiguity and leveraging geometry priors.

Contribution

The paper proposes a new method that integrates monocular depth estimation with NeRF using online training and geometry distillation, improving reconstruction from sparse inputs.

Findings

Achieves state-of-the-art results on real-world datasets.

Robust performance with limited input images.

Effective handling of monocular depth ambiguities.

Abstract

Neural radiance fields (NeRF) shows powerful performance in novel view synthesis and 3D geometry reconstruction, but it suffers from critical performance degradation when the number of known viewpoints is drastically reduced. Existing works attempt to overcome this problem by employing external priors, but their success is limited to certain types of scenes or datasets. Employing monocular depth estimation (MDE) networks, pretrained on large-scale RGB-D datasets, with powerful generalization capability would be a key to solving this problem: however, using MDE in conjunction with NeRF comes with a new set of challenges due to various ambiguity problems exhibited by monocular depths. In this light, we propose a novel framework, dubbed D\"aRF, that achieves robust NeRF reconstruction with a handful of real-world images by combining the strengths of NeRF and monocular depth estimation through online complementary training. Our framework imposes the MDE network's powerful geometry prior to NeRF representation at both seen and unseen viewpoints to enhance its robustness and coherence. In addition, we overcome the ambiguity problems of monocular depths through patch-wise scale-shift fitting and geometry distillation, which adapts the MDE network to produce depths aligned accurately with NeRF geometry. Experiments show our framework achieves state-of-the-art results both quantitatively and qualitatively, demonstrating consistent and reliable performance in both indoor and outdoor real-world datasets. Project page is available at https://ku-cvlab.github.io/DaRF/.