# Unsupervised Cross-spectral Stereo Matching by Learning to Synthesize

## TL;DR

This paper introduces an unsupervised framework for cross-spectral stereo matching that uses style adaptation via cycle consistency and adversarial learning to handle appearance variations, improving disparity estimation without ground truth supervision.

## Contribution

It proposes a novel style adaptation network, F-cycleGAN, for spectral translation, enabling effective unsupervised stereo matching across different spectral images.

## Key findings

- Achieves accurate disparity estimation without ground truth data.
- Effectively minimizes appearance variations between spectral images.
- Enhances robustness of cross-spectral stereo matching.

## Abstract

Unsupervised cross-spectral stereo matching aims at recovering disparity given cross-spectral image pairs without any supervision in the form of ground truth disparity or depth. The estimated depth provides additional information complementary to individual semantic features, which can be helpful for other vision tasks such as tracking, recognition and detection. However, there are large appearance variations between images from different spectral bands, which is a challenge for cross-spectral stereo matching. Existing deep unsupervised stereo matching methods are sensitive to the appearance variations and do not perform well on cross-spectral data. We propose a novel unsupervised cross-spectral stereo matching framework based on image-to-image translation. First, a style adaptation network transforms images across different spectral bands by cycle consistency and adversarial learning, during which appearance variations are minimized. Then, a stereo matching network is trained with image pairs from the same spectra using view reconstruction loss. At last, the estimated disparity is utilized to supervise the spectral-translation network in an end-to-end way. Moreover, a novel style adaptation network F-cycleGAN is proposed to improve the robustness of spectral translation. Our method can tackle appearance variations and enhance the robustness of unsupervised cross-spectral stereo matching. Experimental results show that our method achieves good performance without using depth supervision or explicit semantic information.