Multi-View Stereo with Transformer
Jie Zhu, Bo Peng, Wanqing Li, Haifeng Shen, Zhe Zhang, Jianjun Lei

TL;DR
This paper introduces MVSTR, a Transformer-based network for Multi-View Stereo that leverages global context and 3D geometry to improve dense feature extraction, leading to state-of-the-art results on key datasets.
Contribution
The paper presents a novel Transformer architecture for MVS that enhances feature extraction with global context and 3D consistency, surpassing CNN-based methods.
Findings
Achieves top performance on DTU dataset
Demonstrates strong generalization on Tanks & Temples
Outperforms existing CNN-based MVS methods
Abstract
This paper proposes a network, referred to as MVSTR, for Multi-View Stereo (MVS). It is built upon Transformer and is capable of extracting dense features with global context and 3D consistency, which are crucial to achieving reliable matching for MVS. Specifically, to tackle the problem of the limited receptive field of existing CNN-based MVS methods, a global-context Transformer module is first proposed to explore intra-view global context. In addition, to further enable dense features to be 3D-consistent, a 3D-geometry Transformer module is built with a well-designed cross-view attention mechanism to facilitate inter-view information interaction. Experimental results show that the proposed MVSTR achieves the best overall performance on the DTU dataset and strong generalization on the Tanks & Temples benchmark dataset.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Vision and Imaging · Optical measurement and interference techniques · Advanced Image Processing Techniques
MethodsAttention Is All You Need · Linear Layer · Layer Normalization · Byte Pair Encoding · Label Smoothing · Dense Connections · Absolute Position Encodings · Multi-Head Attention · Residual Connection · Softmax
