Why and How: Knowledge-Guided Learning for Cross-Spectral Image Patch Matching

TL;DR

This paper introduces KGL-Net, a knowledge-guided learning network that bridges descriptor and metric learning for cross-spectral image patch matching, achieving state-of-the-art results without complex architectures.

Contribution

The paper proposes a novel knowledge-guided learning framework and a hard negative sample mining strategy for metric networks, improving cross-spectral patch matching performance.

Findings

Achieves state-of-the-art performance on three cross-spectral matching scenarios.

Introduces a new hard negative sample mining strategy for metric learning.

Demonstrates stability and efficiency with simplified network structures.

Abstract

Recently, cross-spectral image patch matching based on feature relation learning has attracted extensive attention. However, performance bottleneck problems have gradually emerged in existing methods. To address this challenge, we make the first attempt to explore a stable and efficient bridge between descriptor learning and metric learning, and construct a knowledge-guided learning network (KGL-Net), which achieves amazing performance improvements while abandoning complex network structures. Specifically, we find that there is feature extraction consistency between metric learning based on feature difference learning and descriptor learning based on Euclidean distance. This provides the foundation for bridge building. To ensure the stability and efficiency of the constructed bridge, on the one hand, we conduct an in-depth exploration of 20 combined network architectures. On the other hand, a feature-guided loss is constructed to achieve mutual guidance of features. In addition, unlike existing methods, we consider that the feature mapping ability of the metric branch should receive more attention. Therefore, a hard negative sample mining for metric learning (HNSM-M) strategy is constructed. To the best of our knowledge, this is the first time that hard negative sample mining for metric networks has been implemented and brings significant performance gains. Extensive experimental results show that our KGL-Net achieves SOTA performance in three different cross-spectral image patch matching scenarios. Our code are available at https://github.com/YuChuang1205/KGL-Net.