Enhancing Unregistered Hyperspectral Image Super-Resolution via Unmixing-based Abundance Fusion Learning

TL;DR

This paper introduces a novel unmixing-based fusion framework for unregistered hyperspectral image super-resolution, leveraging spectral unmixing, deformable aggregation, and attention mechanisms to improve resolution quality.

Contribution

It proposes a new unmixing-based fusion method that effectively handles unregistered images and enhances super-resolution performance using a multi-stage aggregation and attention approach.

Findings

Achieves state-of-the-art super-resolution results on simulated datasets.

Effectively mitigates unregistered fusion issues with spectral unmixing.

Demonstrates robustness on real hyperspectral datasets.

Abstract

Unregistered hyperspectral image (HSI) super-resolution (SR) typically aims to enhance a low-resolution HSI using an unregistered high-resolution reference image. In this paper, we propose an unmixing-based fusion framework that decouples spatial-spectral information to simultaneously mitigate the impact of unregistered fusion and enhance the learnability of SR models. Specifically, we first utilize singular value decomposition for initial spectral unmixing, preserving the original endmembers while dedicating the subsequent network to enhancing the initial abundance map. To leverage the spatial texture of the unregistered reference, we introduce a coarse-to-fine deformable aggregation module, which first estimates a pixel-level flow and a similarity map using a coarse pyramid predictor. It further performs fine sub-pixel refinement to achieve deformable aggregation of the reference features. The aggregative features are then refined via a series of spatial-channel abundance cross-attention blocks. Furthermore, a spatial-channel modulated fusion module is presented to merge encoder-decoder features using dynamic gating weights, yielding a high-quality, high-resolution HSI. Experimental results on simulated and real datasets confirm that our proposed method achieves state-of-the-art super-resolution performance. The code will be available at https://github.com/yingkai-zhang/UAFL.