Modality-Aware Feature Matching: A Comprehensive Review of Single- and Cross-Modality Techniques

TL;DR

This survey reviews traditional and deep learning-based feature matching techniques across various modalities, emphasizing recent advancements in modality-aware methods for diverse applications like medical imaging and 3D reconstruction.

Contribution

It provides a comprehensive overview of modality-specific feature matching methods, highlighting recent deep learning approaches and their effectiveness across multiple data types.

Findings

Deep learning methods significantly improve robustness across modalities.

Modality-aware descriptors enhance matching accuracy in complex scenarios.

Cross-modal applications benefit from specialized feature matching strategies.

Abstract

Feature matching is a cornerstone task in computer vision, essential for applications such as image retrieval, stereo matching, 3D reconstruction, and SLAM. This survey comprehensively reviews modality-based feature matching, exploring traditional handcrafted methods and emphasizing contemporary deep learning approaches across various modalities, including RGB images, depth images, 3D point clouds, LiDAR scans, medical images, and vision-language interactions. Traditional methods, leveraging detectors like Harris corners and descriptors such as SIFT and ORB, demonstrate robustness under moderate intra-modality variations but struggle with significant modality gaps. Contemporary deep learning-based methods, exemplified by detector-free strategies like CNN-based SuperPoint and transformer-based LoFTR, substantially improve robustness and adaptability across modalities. We highlight modality-aware advancements, such as geometric and depth-specific descriptors for depth images, sparse and dense learning methods for 3D point clouds, attention-enhanced neural networks for LiDAR scans, and specialized solutions like the MIND descriptor for complex medical image matching. Cross-modal applications, particularly in medical image registration and vision-language tasks, underscore the evolution of feature matching to handle increasingly diverse data interactions.