CT-MVSNet: Efficient Multi-View Stereo with Cross-scale Transformer

TL;DR

CT-MVSNet introduces an efficient multi-scale transformer architecture for multi-view stereo that captures intra-image and inter-image features effectively without high computational costs, leading to state-of-the-art results.

Contribution

The paper proposes a novel cross-scale transformer with adaptive matching-aware attention and dual-feature guided aggregation for improved depth estimation in multi-view stereo.

Findings

Achieves state-of-the-art results on DTU and Tanks and Temples datasets.

Reduces computational costs compared to existing transformer-based methods.

Enhances feature representation for more accurate depth estimation.

Abstract

Recent deep multi-view stereo (MVS) methods have widely incorporated transformers into cascade network for high-resolution depth estimation, achieving impressive results. However, existing transformer-based methods are constrained by their computational costs, preventing their extension to finer stages. In this paper, we propose a novel cross-scale transformer (CT) that processes feature representations at different stages without additional computation. Specifically, we introduce an adaptive matching-aware transformer (AMT) that employs different interactive attention combinations at multiple scales. This combined strategy enables our network to capture intra-image context information and enhance inter-image feature relationships. Besides, we present a dual-feature guided aggregation (DFGA) that embeds the coarse global semantic information into the finer cost volume construction to further strengthen global and local feature awareness. Meanwhile, we design a feature metric loss (FM Loss) that evaluates the feature bias before and after transformation to reduce the impact of feature mismatch on depth estimation. Extensive experiments on DTU dataset and Tanks and Temples (T\&T) benchmark demonstrate that our method achieves state-of-the-art results. Code is available at https://github.com/wscstrive/CT-MVSNet.