Learning Robust Stereo Matching in the Wild with Selective Mixture-of-Experts

TL;DR

This paper introduces SMoEStereo, a novel framework that enhances stereo matching robustness across domains by adaptively integrating Vision Foundation Models with mixture-of-experts and low-rank adaptation techniques, balancing accuracy and efficiency.

Contribution

The paper presents a scene-specific fusion approach using MoE and LoRA modules to improve cross-domain stereo matching robustness, with a lightweight decision network for efficiency.

Findings

Achieves state-of-the-art cross-domain performance on multiple benchmarks.

Effectively balances accuracy and computational efficiency.

Demonstrates robustness without dataset-specific tuning.

Abstract

Recently, learning-based stereo matching networks have advanced significantly. However, they often lack robustness and struggle to achieve impressive cross-domain performance due to domain shifts and imbalanced disparity distributions among diverse datasets. Leveraging Vision Foundation Models (VFMs) can intuitively enhance the model's robustness, but integrating such a model into stereo matching cost-effectively to fully realize their robustness remains a key challenge. To address this, we propose SMoEStereo, a novel framework that adapts VFMs for stereo matching through a tailored, scene-specific fusion of Low-Rank Adaptation (LoRA) and Mixture-of-Experts (MoE) modules. SMoEStereo introduces MoE-LoRA with adaptive ranks and MoE-Adapter with adaptive kernel sizes. The former dynamically selects optimal experts within MoE to adapt varying scenes across domains, while the latter injects inductive bias into frozen VFMs to improve geometric feature extraction. Importantly, to mitigate computational overhead, we further propose a lightweight decision network that selectively activates MoE modules based on input complexity, balancing efficiency with accuracy. Extensive experiments demonstrate that our method exhibits state-of-the-art cross-domain and joint generalization across multiple benchmarks without dataset-specific adaptation. The code is available at \textcolor{red}{https://github.com/cocowy1/SMoE-Stereo}.