RobuSTereo: Robust Zero-Shot Stereo Matching under Adverse Weather

TL;DR

RobuSTereo enhances zero-shot stereo matching in adverse weather by generating synthetic training data and designing a robust feature encoder, significantly improving model robustness and generalization in challenging conditions.

Contribution

The paper introduces a diffusion-based data simulation pipeline and a hybrid feature encoder to improve zero-shot stereo matching under adverse weather conditions.

Findings

Significant robustness improvements across diverse weather scenarios.

Reduced domain gap between clean and degraded images.

Enhanced depth estimation accuracy in challenging conditions.

Abstract

Learning-based stereo matching models struggle in adverse weather conditions due to the scarcity of corresponding training data and the challenges in extracting discriminative features from degraded images. These limitations significantly hinder zero-shot generalization to out-of-distribution weather conditions. In this paper, we propose \textbf{RobuSTereo}, a novel framework that enhances the zero-shot generalization of stereo matching models under adverse weather by addressing both data scarcity and feature extraction challenges. First, we introduce a diffusion-based simulation pipeline with a stereo consistency module, which generates high-quality stereo data tailored for adverse conditions. By training stereo matching models on our synthetic datasets, we reduce the domain gap between clean and degraded images, significantly improving the models' robustness to unseen weather conditions. The stereo consistency module ensures structural alignment across synthesized image pairs, preserving geometric integrity and enhancing depth estimation accuracy. Second, we design a robust feature encoder that combines a specialized ConvNet with a denoising transformer to extract stable and reliable features from degraded images. The ConvNet captures fine-grained local structures, while the denoising transformer refines global representations, effectively mitigating the impact of noise, low visibility, and weather-induced distortions. This enables more accurate disparity estimation even under challenging visual conditions. Extensive experiments demonstrate that \textbf{RobuSTereo} significantly improves the robustness and generalization of stereo matching models across diverse adverse weather scenarios.