Blending 3D Geometry and Machine Learning for Multi-View Stereopsis

TL;DR

This paper presents GC MVSNet++, a novel multi-view stereo method that enforces geometric consistency during learning, leading to faster training and state-of-the-art results on multiple datasets.

Contribution

It introduces the first method to incorporate multi-view, multi-scale geometric consistency enforcement during the training process of MVS networks.

Findings

Halves training iterations compared to previous methods.

Achieves state-of-the-art on DTU and BlendedMVS datasets.

Secures second place on Tanks and Temples benchmark.

Abstract

Traditional multi-view stereo (MVS) methods primarily depend on photometric and geometric consistency constraints. In contrast, modern learning-based algorithms often rely on the plane sweep algorithm to infer 3D geometry, applying explicit geometric consistency (GC) checks only as a post-processing step, with no impact on the learning process itself. In this work, we introduce GC MVSNet plus plus, a novel approach that actively enforces geometric consistency of reference view depth maps across multiple source views (multi view) and at various scales (multi scale) during the learning phase (see Fig. 1). This integrated GC check significantly accelerates the learning process by directly penalizing geometrically inconsistent pixels, effectively halving the number of training iterations compared to other MVS methods. Furthermore, we introduce a densely connected cost regularization network with two distinct block designs simple and feature dense optimized to harness dense feature connections for enhanced regularization. Extensive experiments demonstrate that our approach achieves a new state of the art on the DTU and BlendedMVS datasets and secures second place on the Tanks and Temples benchmark. To our knowledge, GC MVSNet plus plus is the first method to enforce multi-view, multi-scale supervised geometric consistency during learning. Our code is available.