Boosting Zero-shot Stereo Matching using Large-scale Mixed Images Sources in the Real World

TL;DR

BooSTer introduces a framework that combines vision foundation models and large-scale mixed image sources, including synthetic and real data, to improve zero-shot stereo matching in real-world scenarios with limited labeled data.

Contribution

The paper presents a novel approach that leverages monocular depth estimation, diffusion models, and foundation models to enhance zero-shot stereo matching performance.

Findings

Significant accuracy improvements over existing methods.

Effective handling of domain gaps between synthetic and real images.

Robust performance with limited labeled data.

Abstract

Stereo matching methods rely on dense pixel-wise ground truth labels, which are laborious to obtain, especially for real-world datasets. The scarcity of labeled data and domain gaps between synthetic and real-world images also pose notable challenges. In this paper, we propose a novel framework, \textbf{BooSTer}, that leverages both vision foundation models and large-scale mixed image sources, including synthetic, real, and single-view images. First, to fully unleash the potential of large-scale single-view images, we design a data generation strategy combining monocular depth estimation and diffusion models to generate dense stereo matching data from single-view images. Second, to tackle sparse labels in real-world datasets, we transfer knowledge from monocular depth estimation models, using pseudo-mono depth labels and a dynamic scale- and shift-invariant loss for additional supervision. Furthermore, we incorporate vision foundation model as an encoder to extract robust and transferable features, boosting accuracy and generalization. Extensive experiments on benchmark datasets demonstrate the effectiveness of our approach, achieving significant improvements in accuracy over existing methods, particularly in scenarios with limited labeled data and domain shifts.