ConsistentNeRF: Enhancing Neural Radiance Fields with 3D Consistency for Sparse View Synthesis

TL;DR

ConsistentNeRF improves neural radiance field reconstructions from sparse views by enforcing 3D pixel consistency using depth information, leading to significant quality enhancements across multiple benchmarks.

Contribution

It introduces a depth-based regularization method that enhances 3D consistency in NeRFs, especially under sparse view conditions, which was not addressed in prior work.

Findings

Up to 94% PSNR improvement over baseline

76% SSIM improvement over baseline

31% LPIPS reduction over baseline

Abstract

Neural Radiance Fields (NeRF) has demonstrated remarkable 3D reconstruction capabilities with dense view images. However, its performance significantly deteriorates under sparse view settings. We observe that learning the 3D consistency of pixels among different views is crucial for improving reconstruction quality in such cases. In this paper, we propose ConsistentNeRF, a method that leverages depth information to regularize both multi-view and single-view 3D consistency among pixels. Specifically, ConsistentNeRF employs depth-derived geometry information and a depth-invariant loss to concentrate on pixels that exhibit 3D correspondence and maintain consistent depth relationships. Extensive experiments on recent representative works reveal that our approach can considerably enhance model performance in sparse view conditions, achieving improvements of up to 94% in PSNR, 76% in SSIM, and 31% in LPIPS compared to the vanilla baselines across various benchmarks, including DTU, NeRF Synthetic, and LLFF.