Fourier-enhanced Implicit Neural Fusion Network for Multispectral and Hyperspectral Image Fusion

TL;DR

This paper introduces FeINFN, a novel neural network architecture enhanced with Fourier and Gabor wavelet techniques, to improve high-frequency detail preservation and global perception in multispectral and hyperspectral image fusion tasks, achieving state-of-the-art results.

Contribution

The paper proposes a Fourier-enhanced implicit neural fusion network with a novel fusion function and a Gabor wavelet-based decoder, addressing high-frequency loss and limited perceptual scope in INR for MHIF.

Findings

Achieves state-of-the-art performance on benchmark datasets.

Effectively captures high-frequency information and expands receptive field.

Demonstrates the benefits of Fourier and Gabor wavelet integration in INR.

Abstract

Recently, implicit neural representations (INR) have made significant strides in various vision-related domains, providing a novel solution for Multispectral and Hyperspectral Image Fusion (MHIF) tasks. However, INR is prone to losing high-frequency information and is confined to the lack of global perceptual capabilities. To address these issues, this paper introduces a Fourier-enhanced Implicit Neural Fusion Network (FeINFN) specifically designed for MHIF task, targeting the following phenomena: The Fourier amplitudes of the HR-HSI latent code and LR-HSI are remarkably similar; however, their phases exhibit different patterns. In FeINFN, we innovatively propose a spatial and frequency implicit fusion function (Spa-Fre IFF), helping INR capture high-frequency information and expanding the receptive field. Besides, a new decoder employing a complex Gabor wavelet activation function, called Spatial-Frequency Interactive Decoder (SFID), is invented to enhance the interaction of INR features. Especially, we further theoretically prove that the Gabor wavelet activation possesses a time-frequency tightness property that favors learning the optimal bandwidths in the decoder. Experiments on two benchmark MHIF datasets verify the state-of-the-art (SOTA) performance of the proposed method, both visually and quantitatively. Also, ablation studies demonstrate the mentioned contributions. The code will be available on Anonymous GitHub (https://anonymous.4open.science/r/FeINFN-15C9/) after possible acceptance.