DiViNeT: 3D Reconstruction from Disparate Views via Neural Template Regularization

TL;DR

DiViNeT introduces a neural template regularization approach for 3D surface reconstruction from as few as three disparate RGB images, effectively completing and detailing surfaces despite sparse views.

Contribution

The paper proposes a novel two-stage neural template regularization method that learns surface priors without supervision and improves 3D reconstruction from sparse views.

Findings

Achieves state-of-the-art reconstruction quality on DTU and BlendedMVS datasets with sparse views.

Performs comparably or better than existing methods with dense views.

Successfully reconstructs surface details and completes geometry from limited input images.

Abstract

We present a volume rendering-based neural surface reconstruction method that takes as few as three disparate RGB images as input. Our key idea is to regularize the reconstruction, which is severely ill-posed and leaving significant gaps between the sparse views, by learning a set of neural templates to act as surface priors. Our method, coined DiViNet, operates in two stages. It first learns the templates, in the form of 3D Gaussian functions, across different scenes, without 3D supervision. In the reconstruction stage, our predicted templates serve as anchors to help "stitch'' the surfaces over sparse regions. We demonstrate that our approach is not only able to complete the surface geometry but also reconstructs surface details to a reasonable extent from a few disparate input views. On the DTU and BlendedMVS datasets, our approach achieves the best reconstruction quality among existing methods in the presence of such sparse views and performs on par, if not better, with competing methods when dense views are employed as inputs.