MVSFormer++: Revealing the Devil in Transformer's Details for Multi-View Stereo

TL;DR

MVSFormer++ enhances multi-view stereo by leveraging transformer attention mechanisms and design optimizations, achieving state-of-the-art results across multiple benchmarks.

Contribution

The paper introduces MVSFormer++, which maximizes transformer attention characteristics and incorporates novel design details to improve depth estimation in MVS tasks.

Findings

Achieves state-of-the-art performance on DTU and Tanks-and-Temples benchmarks.

Effectively integrates cross-view information using DINOv2.

Highlights importance of design details like positional encoding and normalization.

Abstract

Recent advancements in learning-based Multi-View Stereo (MVS) methods have prominently featured transformer-based models with attention mechanisms. However, existing approaches have not thoroughly investigated the profound influence of transformers on different MVS modules, resulting in limited depth estimation capabilities. In this paper, we introduce MVSFormer++, a method that prudently maximizes the inherent characteristics of attention to enhance various components of the MVS pipeline. Formally, our approach involves infusing cross-view information into the pre-trained DINOv2 model to facilitate MVS learning. Furthermore, we employ different attention mechanisms for the feature encoder and cost volume regularization, focusing on feature and spatial aggregations respectively. Additionally, we uncover that some design details would substantially impact the performance of transformer modules in MVS, including normalized 3D positional encoding, adaptive attention scaling, and the position of layer normalization. Comprehensive experiments on DTU, Tanks-and-Temples, BlendedMVS, and ETH3D validate the effectiveness of the proposed method. Notably, MVSFormer++ achieves state-of-the-art performance on the challenging DTU and Tanks-and-Temples benchmarks.