RGB-Multispectral Matching: Dataset, Learning Methodology, Evaluation

TL;DR

This paper introduces a new dataset and a deep learning approach for registering RGB and multi-spectral images with different resolutions, achieving promising accuracy in a challenging cross-modal matching task.

Contribution

The paper presents a novel RGB-MS dataset with high-resolution disparity ground-truth and a self-supervised learning method leveraging RGB-RGB matching to improve cross-modal registration.

Findings

Achieved 1.16 pixels average registration error

Created a dataset with 34 annotated image pairs in indoor scenes

Demonstrated effective self-supervised training without ground-truth labels

Abstract

We address the problem of registering synchronized color (RGB) and multi-spectral (MS) images featuring very different resolution by solving stereo matching correspondences. Purposely, we introduce a novel RGB-MS dataset framing 13 different scenes in indoor environments and providing a total of 34 image pairs annotated with semi-dense, high-resolution ground-truth labels in the form of disparity maps. To tackle the task, we propose a deep learning architecture trained in a self-supervised manner by exploiting a further RGB camera, required only during training data acquisition. In this setup, we can conveniently learn cross-modal matching in the absence of ground-truth labels by distilling knowledge from an easier RGB-RGB matching task based on a collection of about 11K unlabeled image triplets. Experiments show that the proposed pipeline sets a good performance bar (1.16 pixels average registration error) for future research on this novel, challenging task.