SMLNet: A SPD Manifold Learning Network for Infrared and Visible Image Fusion

TL;DR

SMLNet introduces a novel SPD manifold learning approach for infrared and visible image fusion, leveraging Riemannian geometry to better capture intrinsic correlations and improve fusion quality over traditional Euclidean methods.

Contribution

The paper proposes a new SPD manifold learning framework for multi-modal image fusion, extending from Euclidean space to Riemannian geometry to enhance feature representation.

Findings

Outperforms state-of-the-art methods on public datasets

Effectively captures intrinsic statistical correlations in images

Improves fusion quality by leveraging SPD manifold structure

Abstract

Euclidean representation learning methods have achieved promising results in image fusion tasks, which can be attributed to their clear advantages in handling with linear space. However, data collected from a realistic scene usually has a non-Euclidean structure, evaluating the consistency of latent representations from paired views using Euclidean distance raises challenges. To address this issue, a novel SPD (symmetric positive definite) manifold learning is proposed for multi-modal image fusion, named SMLNet, which extends the image fusion approach from the Euclidean space to the SPD manifolds. Specifically, we encode images according to the Riemannian geometry to exploit their intrinsic statistical correlations, thereby aligning with human visual perception. The SPD matrix fundamentally underpins our network's learning process. Building upon this mathematical foundation, we employ a cross-modal fusion strategy to exploit modality-specific dependencies and augment complementary information. To capture semantic similarity in images' intrinsic space, we further develop an attention module that meticulously processes the cross-modal semantic affinity matrix. Based on this, we design an end-to-end fusion network based on cross-modal manifold learning. Extensive experiments on public datasets demonstrate that our framework exhibits superior performance compared to the current state-of-the-art methods. Our code will be publicly available at https://github.com/Shaoyun2023.