SurfaceNet+: An End-to-end 3D Neural Network for Very Sparse Multi-view Stereopsis

TL;DR

SurfaceNet+ is an end-to-end 3D neural network designed for very sparse multi-view stereopsis, effectively handling large baseline angles and extreme sparsity by novel view selection and multi-scale refinement, outperforming existing methods.

Contribution

The paper introduces SurfaceNet+, a volumetric approach with a novel view selection and multi-scale refinement to improve 3D reconstruction in extremely sparse multi-view scenarios.

Findings

SurfaceNet+ significantly outperforms state-of-the-art methods in sparse MVS accuracy.

It maintains high-quality 3D reconstruction even with very few views.

The method is effective under large baseline angles and extreme sparsity conditions.

Abstract

Multi-view stereopsis (MVS) tries to recover the 3D model from 2D images. As the observations become sparser, the significant 3D information loss makes the MVS problem more challenging. Instead of only focusing on densely sampled conditions, we investigate sparse-MVS with large baseline angles since the sparser sensation is more practical and more cost-efficient. By investigating various observation sparsities, we show that the classical depth-fusion pipeline becomes powerless for the case with a larger baseline angle that worsens the photo-consistency check. As another line of the solution, we present SurfaceNet+, a volumetric method to handle the 'incompleteness' and the 'inaccuracy' problems induced by a very sparse MVS setup. Specifically, the former problem is handled by a novel volume-wise view selection approach. It owns superiority in selecting valid views while discarding invalid occluded views by considering the geometric prior. Furthermore, the latter problem is handled via a multi-scale strategy that consequently refines the recovered geometry around the region with the repeating pattern. The experiments demonstrate the tremendous performance gap between SurfaceNet+ and state-of-the-art methods in terms of precision and recall. Under the extreme sparse-MVS settings in two datasets, where existing methods can only return very few points, SurfaceNet+ still works as well as in the dense MVS setting. The benchmark and the implementation are publicly available at https://github.com/mjiUST/SurfaceNet-plus.