MIFNet: Learning Modality-Invariant Features for Generalizable Multimodal Image Matching

TL;DR

MIFNet is a novel neural network that learns modality-invariant features for multimodal image matching using only single-modality training data, enabling robust zero-shot performance across diverse modalities.

Contribution

The paper introduces MIFNet, a new approach that leverages pre-trained features and novel modules to learn modality-invariant descriptors without requiring multimodal training data.

Findings

MIFNet achieves strong zero-shot generalization on multiple multimodal datasets.

The method outperforms existing approaches in multimodal image matching tasks.

It effectively leverages pre-trained models to enhance robustness across modalities.

Abstract

Many keypoint detection and description methods have been proposed for image matching or registration. While these methods demonstrate promising performance for single-modality image matching, they often struggle with multimodal data because the descriptors trained on single-modality data tend to lack robustness against the non-linear variations present in multimodal data. Extending such methods to multimodal image matching often requires well-aligned multimodal data to learn modality-invariant descriptors. However, acquiring such data is often costly and impractical in many real-world scenarios. To address this challenge, we propose a modality-invariant feature learning network (MIFNet) to compute modality-invariant features for keypoint descriptions in multimodal image matching using only single-modality training data. Specifically, we propose a novel latent feature aggregation module and a cumulative hybrid aggregation module to enhance the base keypoint descriptors trained on single-modality data by leveraging pre-trained features from Stable Diffusion models. %, our approach generates robust and invariant features across diverse and unknown modalities. We validate our method with recent keypoint detection and description methods in three multimodal retinal image datasets (CF-FA, CF-OCT, EMA-OCTA) and two remote sensing datasets (Optical-SAR and Optical-NIR). Extensive experiments demonstrate that the proposed MIFNet is able to learn modality-invariant feature for multimodal image matching without accessing the targeted modality and has good zero-shot generalization ability. The code will be released at https://github.com/lyp-deeplearning/MIFNet.