UniTT-Stereo: Unified Training of Transformer for Enhanced Stereo Matching

TL;DR

UniTT-Stereo introduces a unified self-supervised and supervised training framework for Transformer-based stereo matching, significantly improving performance on multiple benchmarks by leveraging feature reconstruction and adaptive masking strategies.

Contribution

It is the first to unify self-supervised pre-training with supervised stereo matching training for Transformer architectures, enhancing data efficiency and accuracy.

Findings

Achieves state-of-the-art results on ETH3D, KITTI 2012, and KITTI 2015 datasets.

Demonstrates the effectiveness of feature reconstruction and adaptive masking in limited data scenarios.

Provides insights into the locality inductive bias through frequency and attention map analysis.

Abstract

Unlike other vision tasks where Transformer-based approaches are becoming increasingly common, stereo depth estimation is still dominated by convolution-based approaches. This is mainly due to the limited availability of real-world ground truth for stereo matching, which is a limiting factor in improving the performance of Transformer-based stereo approaches. In this paper, we propose UniTT-Stereo, a method to maximize the potential of Transformer-based stereo architectures by unifying self-supervised learning used for pre-training with stereo matching framework based on supervised learning. To be specific, we explore the effectiveness of reconstructing features of masked portions in an input image and at the same time predicting corresponding points in another image from the perspective of locality inductive bias, which is crucial in training models with limited training data. Moreover, to address these challenging tasks of reconstruction-and-prediction, we present a new strategy to vary a masking ratio when training the stereo model with stereo-tailored losses. State-of-the-art performance of UniTT-Stereo is validated on various benchmarks such as ETH3D, KITTI 2012, and KITTI 2015 datasets. Lastly, to investigate the advantages of the proposed approach, we provide a frequency analysis of feature maps and the analysis of locality inductive bias based on attention maps.