PIF-Net: Ill-Posed Prior Guided Multispectral and Hyperspectral Image Fusion via Invertible Mamba and Fusion-Aware LoRA

TL;DR

PIF-Net is a novel fusion framework that effectively addresses the ill-posed nature of multispectral and hyperspectral image fusion by incorporating priors, using invertible architecture, and a fusion-aware adaptation module, leading to superior results.

Contribution

The paper introduces PIF-Net, a new fusion framework combining an invertible Mamba architecture and a fusion-aware LoRA module to improve spectral and spatial image fusion.

Findings

Outperforms state-of-the-art methods on benchmark datasets

Maintains high spectral and spatial fidelity with efficient computation

Demonstrates robustness to data misalignment and limited observations

Abstract

The goal of multispectral and hyperspectral image fusion (MHIF) is to generate high-quality images that simultaneously possess rich spectral information and fine spatial details. However, due to the inherent trade-off between spectral and spatial information and the limited availability of observations, this task is fundamentally ill-posed. Previous studies have not effectively addressed the ill-posed nature caused by data misalignment. To tackle this challenge, we propose a fusion framework named PIF-Net, which explicitly incorporates ill-posed priors to effectively fuse multispectral images and hyperspectral images. To balance global spectral modeling with computational efficiency, we design a method based on an invertible Mamba architecture that maintains information consistency during feature transformation and fusion, ensuring stable gradient flow and process reversibility. Furthermore, we introduce a novel fusion module called the Fusion-Aware Low-Rank Adaptation module, which dynamically calibrates spectral and spatial features while keeping the model lightweight. Extensive experiments on multiple benchmark datasets demonstrate that PIF-Net achieves significantly better image restoration performance than current state-of-the-art methods while maintaining model efficiency.