Enhancing Neural Radiance Fields with Depth and Normal Completion Priors from Sparse Views

TL;DR

This paper introduces CP_NeRF, a method that enhances Neural Radiance Fields by integrating depth and normal completion priors derived from sparse views, significantly improving rendering quality in limited-view scenarios.

Contribution

The paper proposes a novel framework that incorporates dense depth and normal priors into NeRF optimization, using sparse data and patch matching to improve scene rendering accuracy.

Findings

Outperforms existing methods in indoor scene rendering with sparse views.

Effectively transforms sparse depth and normal data into dense priors for better guidance.

Achieves more accurate normal and depth estimation, leading to improved visual quality.

Abstract

Neural Radiance Fields (NeRF) are an advanced technology that creates highly realistic images by learning about scenes through a neural network model. However, NeRF often encounters issues when there are not enough images to work with, leading to problems in accurately rendering views. The main issue is that NeRF lacks sufficient structural details to guide the rendering process accurately. To address this, we proposed a Depth and Normal Dense Completion Priors for NeRF (CP\_NeRF) framework. This framework enhances view rendering by adding depth and normal dense completion priors to the NeRF optimization process. Before optimizing NeRF, we obtain sparse depth maps using the Structure from Motion (SfM) technique used to get camera poses. Based on the sparse depth maps and a normal estimator, we generate sparse normal maps for training a normal completion prior with precise standard deviations. During optimization, we apply depth and normal completion priors to transform sparse data into dense depth and normal maps with their standard deviations. We use these dense maps to guide ray sampling, assist distance sampling and construct a normal loss function for better training accuracy. To improve the rendering of NeRF's normal outputs, we incorporate an optical centre position embedder that helps synthesize more accurate normals through volume rendering. Additionally, we employ a normal patch matching technique to choose accurate rendered normal maps, ensuring more precise supervision for the model. Our method is superior to leading techniques in rendering detailed indoor scenes, even with limited input views.