Implicit Neural Feature Fusion Function for Multispectral and Hyperspectral Image Fusion

TL;DR

This paper introduces a novel INR-based hyperspectral image fusion method called INF, which effectively combines high-frequency details from multispectral and hyperspectral images to produce high-resolution hyperspectral images with state-of-the-art results.

Contribution

The paper proposes a new Implicit Neural Feature Fusion (INF) method with dual high-frequency fusion and a parameter-free cosine similarity approach, improving interpretability and performance in MHIF.

Findings

Achieves state-of-the-art results on CAVE and Harvard datasets.

Effectively fuses high-frequency details from multispectral and hyperspectral images.

Demonstrates improved interpretability over CNN-based methods.

Abstract

Multispectral and Hyperspectral Image Fusion (MHIF) is a practical task that aims to fuse a high-resolution multispectral image (HR-MSI) and a low-resolution hyperspectral image (LR-HSI) of the same scene to obtain a high-resolution hyperspectral image (HR-HSI). Benefiting from powerful inductive bias capability, CNN-based methods have achieved great success in the MHIF task. However, they lack certain interpretability and require convolution structures be stacked to enhance performance. Recently, Implicit Neural Representation (INR) has achieved good performance and interpretability in 2D tasks due to its ability to locally interpolate samples and utilize multimodal content such as pixels and coordinates. Although INR-based approaches show promise, they require extra construction of high-frequency information (\emph{e.g.,} positional encoding). In this paper, inspired by previous work of MHIF task, we realize that HR-MSI could serve as a high-frequency detail auxiliary input, leading us to propose a novel INR-based hyperspectral fusion function named Implicit Neural Feature Fusion Function (INF). As an elaborate structure, it solves the MHIF task and addresses deficiencies in the INR-based approaches. Specifically, our INF designs a Dual High-Frequency Fusion (DHFF) structure that obtains high-frequency information twice from HR-MSI and LR-HSI, then subtly fuses them with coordinate information. Moreover, the proposed INF incorporates a parameter-free method named INR with cosine similarity (INR-CS) that uses cosine similarity to generate local weights through feature vectors. Based on INF, we construct an Implicit Neural Fusion Network (INFN) that achieves state-of-the-art performance for MHIF tasks of two public datasets, \emph{i.e.,} CAVE and Harvard. The code will soon be made available on GitHub.