SCADE: NeRFs from Space Carving with Ambiguity-Aware Depth Estimates

TL;DR

SCADE enhances NeRF-based 3D scene reconstruction from limited views by integrating ambiguity-aware depth priors and a space carving loss, leading to improved fidelity in sparse-view indoor scene synthesis.

Contribution

The paper introduces a novel method combining multimodal depth estimation with space carving to improve NeRF reconstructions from sparse views.

Findings

Achieves higher fidelity novel view synthesis with fewer input views.

Effectively handles depth ambiguities using conditional IMLE.

Demonstrates robustness on in-the-wild indoor scenes.

Abstract

Neural radiance fields (NeRFs) have enabled high fidelity 3D reconstruction from multiple 2D input views. However, a well-known drawback of NeRFs is the less-than-ideal performance under a small number of views, due to insufficient constraints enforced by volumetric rendering. To address this issue, we introduce SCADE, a novel technique that improves NeRF reconstruction quality on sparse, unconstrained input views for in-the-wild indoor scenes. To constrain NeRF reconstruction, we leverage geometric priors in the form of per-view depth estimates produced with state-of-the-art monocular depth estimation models, which can generalize across scenes. A key challenge is that monocular depth estimation is an ill-posed problem, with inherent ambiguities. To handle this issue, we propose a new method that learns to predict, for each view, a continuous, multimodal distribution of depth estimates using conditional Implicit Maximum Likelihood Estimation (cIMLE). In order to disambiguate exploiting multiple views, we introduce an original space carving loss that guides the NeRF representation to fuse multiple hypothesized depth maps from each view and distill from them a common geometry that is consistent with all views. Experiments show that our approach enables higher fidelity novel view synthesis from sparse views. Our project page can be found at https://scade-spacecarving-nerfs.github.io .