Hyperspectral and multispectral image fusion with arbitrary resolution through self-supervised representations

TL;DR

This paper introduces a self-supervised neural method for hyperspectral and multispectral image fusion that allows flexible resolution adjustments without retraining, outperforming existing techniques.

Contribution

The proposed continuous low-rank factorization (CLoRF) enables resolution-independent fusion by integrating neural representations with matrix factorization, a novel approach in hyperspectral image super-resolution.

Findings

Significantly outperforms existing fusion methods.

Achieves user-desired resolutions without retraining.

Proves low-rank and Lipschitz properties theoretically.

Abstract

The fusion of a low-resolution hyperspectral image (LR-HSI) with a high-resolution multispectral image (HR-MSI) has emerged as an effective technique for achieving HSI super-resolution (SR). Previous studies have mainly concentrated on estimating the posterior distribution of the latent high-resolution hyperspectral image (HR-HSI), leveraging an appropriate image prior and likelihood computed from the discrepancy between the latent HSI and observed images. Low rankness stands out for preserving latent HSI characteristics through matrix factorization among the various priors. However, the primary limitation in previous studies lies in the generalization of a fusion model with fixed resolution scales, which necessitates retraining whenever output resolutions are changed. To overcome this limitation, we propose a novel continuous low-rank factorization (CLoRF) by integrating two neural representations into the matrix factorization, capturing spatial and spectral information, respectively. This approach enables us to harness both the low rankness from the matrix factorization and the continuity from neural representation in a self-supervised manner.Theoretically, we prove the low-rank property and Lipschitz continuity in the proposed continuous low-rank factorization. Experimentally, our method significantly surpasses existing techniques and achieves user-desired resolutions without the need for neural network retraining. Code is available at https://github.com/wangting1907/CLoRF-Fusion.