Multi-View Stereo with Transformer

TL;DR

This paper introduces MVSTR, a Transformer-based network for Multi-View Stereo that leverages global context and 3D geometry to improve dense feature extraction, leading to state-of-the-art results on key datasets.

Contribution

The paper presents a novel Transformer architecture for MVS that enhances feature extraction with global context and 3D consistency, surpassing CNN-based methods.

Findings

Achieves top performance on DTU dataset

Demonstrates strong generalization on Tanks & Temples

Outperforms existing CNN-based MVS methods

Abstract

This paper proposes a network, referred to as MVSTR, for Multi-View Stereo (MVS). It is built upon Transformer and is capable of extracting dense features with global context and 3D consistency, which are crucial to achieving reliable matching for MVS. Specifically, to tackle the problem of the limited receptive field of existing CNN-based MVS methods, a global-context Transformer module is first proposed to explore intra-view global context. In addition, to further enable dense features to be 3D-consistent, a 3D-geometry Transformer module is built with a well-designed cross-view attention mechanism to facilitate inter-view information interaction. Experimental results show that the proposed MVSTR achieves the best overall performance on the DTU dataset and strong generalization on the Tanks & Temples benchmark dataset.