Rethinking the Key Factors for the Generalization of Remote Sensing Stereo Matching Networks

TL;DR

This paper investigates key factors affecting the generalization of remote sensing stereo matching networks, emphasizing dataset selection, model structure, and training methods, and proposes an unsupervised network with improved cross-domain performance.

Contribution

It identifies critical training factors for better generalization and introduces an unsupervised stereo matching network tailored for remote sensing data.

Findings

Unsupervised methods outperform supervised ones in cross-domain generalization.

Selecting training data with similar regional distribution improves performance.

A cascaded model structure enhances adaptability to different feature sizes.

Abstract

Stereo matching, a critical step of 3D reconstruction, has fully shifted towards deep learning due to its strong feature representation of remote sensing images. However, ground truth for stereo matching task relies on expensive airborne LiDAR data, thus making it difficult to obtain enough samples for supervised learning. To improve the generalization ability of stereo matching networks on cross-domain data from different sensors and scenarios, in this paper, we dedicate to study key training factors from three perspectives. (1) For the selection of training dataset, it is important to select data with similar regional target distribution as the test set instead of utilizing data from the same sensor. (2) For model structure, cascaded structure that flexibly adapts to different sizes of features is preferred. (3) For training manner, unsupervised methods generalize better than supervised methods, and we design an unsupervised early-stop strategy to help retain the best model with pre-trained weights as the basis. Extensive experiments are conducted to support the previous findings, on the basis of which we present an unsupervised stereo matching network with good generalization performance. We release the source code and the datasets at https://github.com/Elenairene/RKF_RSSM to reproduce the results and encourage future work.