RIAV-MVS: Recurrent-Indexing an Asymmetric Volume for Multi-View Stereo

TL;DR

This paper introduces RIAV-MVS, a novel learning-based multi-view stereo method that uses an asymmetric volume with transformer and residual pose correction to improve depth estimation accuracy and generalization.

Contribution

The paper proposes an innovative asymmetric volume construction with transformer blocks and residual pose correction, enhancing multi-view stereo depth estimation.

Findings

Achieves state-of-the-art results on real-world MVS datasets.

Demonstrates strong cross-dataset generalization.

Outperforms existing methods in accuracy and robustness.

Abstract

This paper presents a learning-based method for multi-view depth estimation from posed images. Our core idea is a "learning-to-optimize" paradigm that iteratively indexes a plane-sweeping cost volume and regresses the depth map via a convolutional Gated Recurrent Unit (GRU). Since the cost volume plays a paramount role in encoding the multi-view geometry, we aim to improve its construction both at pixel- and frame- levels. At the pixel level, we propose to break the symmetry of the Siamese network (which is typically used in MVS to extract image features) by introducing a transformer block to the reference image (but not to the source images). Such an asymmetric volume allows the network to extract global features from the reference image to predict its depth map. Given potential inaccuracies in the poses between reference and source images, we propose to incorporate a residual pose network to correct the relative poses. This essentially rectifies the cost volume at the frame level. We conduct extensive experiments on real-world MVS datasets and show that our method achieves state-of-the-art performance in terms of both within-dataset evaluation and cross-dataset generalization. Code available: https://github.com/oppo-us-research/riav-mvs.