Unregistered Spectral Image Fusion: Unmixing, Adversarial Learning, and Recoverability

TL;DR

This paper introduces an unsupervised spectral image fusion method that unmixes, learns adversarially, and guarantees recoverability of super-resolved hyperspectral and multispectral images in unregistered settings.

Contribution

It presents the first theoretical and practical framework for unregistered hyperspectral-multispectral fusion with recoverability guarantees and unsupervised learning.

Findings

Method achieves high-quality super-resolution on real HSI-MSI pairs.

Theoretical guarantees ensure recoverability under certain models.

Validated across diverse semi-real and real datasets.

Abstract

This paper addresses the fusion of a pair of spatially unregistered hyperspectral image (HSI) and multispectral image (MSI) covering roughly overlapping regions. HSIs offer high spectral but low spatial resolution, while MSIs provide the opposite. The goal is to integrate their complementary information to enhance both HSI spatial resolution and MSI spectral resolution. While hyperspectral-multispectral fusion (HMF) has been widely studied, the unregistered setting remains challenging. Many existing methods focus solely on MSI super-resolution, leaving HSI unchanged. Supervised deep learning approaches were proposed for HSI super-resolution, but rely on accurate training data, which is often unavailable. Moreover, theoretical analyses largely address the co-registered case, leaving unregistered HMF poorly understood. In this work, an unsupervised framework is proposed to simultaneously super-resolve both MSI and HSI. The method integrates coupled spectral unmixing for MSI super-resolution with latent-space adversarial learning for HSI super-resolution. Theoretical guarantees on the recoverability of the super-resolution MSI and HSI are established under reasonable generative models -- providing, to our best knowledge, the first such insights for unregistered HMF. The approach is validated on semi-real and real HSI-MSI pairs across diverse conditions.