Depth-supervised NeRF: Fewer Views and Faster Training for Free

TL;DR

DS-NeRF introduces a depth supervision loss that leverages sparse 3D points from structure-from-motion to improve NeRF training, enabling fewer views, faster training, and better rendering quality.

Contribution

The paper proposes DS-NeRF, a novel depth supervision loss that enhances NeRF training by utilizing readily available sparse 3D points from SFM, improving efficiency and quality.

Findings

Fewer views needed for high-quality rendering.

Training speed increased by 2-3 times.

Compatible with various depth supervision sources.

Abstract

A commonly observed failure mode of Neural Radiance Field (NeRF) is fitting incorrect geometries when given an insufficient number of input views. One potential reason is that standard volumetric rendering does not enforce the constraint that most of a scene's geometry consist of empty space and opaque surfaces. We formalize the above assumption through DS-NeRF (Depth-supervised Neural Radiance Fields), a loss for learning radiance fields that takes advantage of readily-available depth supervision. We leverage the fact that current NeRF pipelines require images with known camera poses that are typically estimated by running structure-from-motion (SFM). Crucially, SFM also produces sparse 3D points that can be used as "free" depth supervision during training: we add a loss to encourage the distribution of a ray's terminating depth matches a given 3D keypoint, incorporating depth uncertainty. DS-NeRF can render better images given fewer training views while training 2-3x faster. Further, we show that our loss is compatible with other recently proposed NeRF methods, demonstrating that depth is a cheap and easily digestible supervisory signal. And finally, we find that DS-NeRF can support other types of depth supervision such as scanned depth sensors and RGB-D reconstruction outputs.