There and Back Again: Self-supervised Multispectral Correspondence Estimation

TL;DR

This paper introduces a spectra-agnostic, self-supervised method for dense correspondence estimation across multiple spectral images, demonstrating improved accuracy in challenging cross-spectral tasks like RGB-FIR, RGB-NIR, and RGB-RGB.

Contribution

It proposes a novel cycle-consistency based self-supervision framework that generalizes dense correspondence estimation across different spectra without spectrum-specific tuning.

Findings

Achieves higher accuracy than similar self-supervised methods on RGB-FIR, RGB-NIR, and RGB-RGB tasks.

Demonstrates the effectiveness of a spectra-agnostic, unified approach for cross-spectral correspondence.

Validates the generalization capability of the proposed method across multiple spectral pairs.

Abstract

Across a wide range of applications, from autonomous vehicles to medical imaging, multi-spectral images provide an opportunity to extract additional information not present in color images. One of the most important steps in making this information readily available is the accurate estimation of dense correspondences between different spectra. Due to the nature of cross-spectral images, most correspondence solving techniques for the visual domain are simply not applicable. Furthermore, most cross-spectral techniques utilize spectra-specific characteristics to perform the alignment. In this work, we aim to address the dense correspondence estimation problem in a way that generalizes to more than one spectrum. We do this by introducing a novel cycle-consistency metric that allows us to self-supervise. This, combined with our spectra-agnostic loss functions, allows us to train the same network across multiple spectra. We demonstrate our approach on the challenging task of dense RGB-FIR correspondence estimation. We also show the performance of our unmodified network on the cases of RGB-NIR and RGB-RGB, where we achieve higher accuracy than similar self-supervised approaches. Our work shows that cross-spectral correspondence estimation can be solved in a common framework that learns to generalize alignment across spectra.