Learning Neural Radiance Fields from Multi-View Geometry

TL;DR

This paper introduces MVG-NeRF, a framework that integrates classical multi-view geometry with neural radiance fields to improve 3D reconstruction quality by using geometric priors and confidence weighting.

Contribution

It proposes leveraging pixelwise depths and normals as priors in NeRF training, enhancing surface accuracy and mesh quality in 3D reconstruction from images.

Findings

Improved mesh quality with cleaner 3D surfaces.

Maintains competitive novel view synthesis performance.

Robustness enhanced by confidence-weighted pixel contributions.

Abstract

We present a framework, called MVG-NeRF, that combines classical Multi-View Geometry algorithms and Neural Radiance Fields (NeRF) for image-based 3D reconstruction. NeRF has revolutionized the field of implicit 3D representations, mainly due to a differentiable volumetric rendering formulation that enables high-quality and geometry-aware novel view synthesis. However, the underlying geometry of the scene is not explicitly constrained during training, thus leading to noisy and incorrect results when extracting a mesh with marching cubes. To this end, we propose to leverage pixelwise depths and normals from a classical 3D reconstruction pipeline as geometric priors to guide NeRF optimization. Such priors are used as pseudo-ground truth during training in order to improve the quality of the estimated underlying surface. Moreover, each pixel is weighted by a confidence value based on the forward-backward reprojection error for additional robustness. Experimental results on real-world data demonstrate the effectiveness of this approach in obtaining clean 3D meshes from images, while maintaining competitive performances in novel view synthesis.