Multispectral and Hyperspectral Image Fusion by MS/HS Fusion Net

TL;DR

This paper introduces MS/HS Fusion Net, a deep learning model that effectively merges high-resolution multispectral and low-resolution hyperspectral images to produce high-resolution hyperspectral images, outperforming existing methods.

Contribution

The paper presents a novel model-based deep learning approach that incorporates observation models and spectral low-rankness, with an unfolded iterative algorithm into a trainable neural network.

Findings

Outperforms state-of-the-art methods visually and quantitatively

Effective on both simulated and real data

Demonstrates superior spectral and spatial resolution fusion

Abstract

Hyperspectral imaging can help better understand the characteristics of different materials, compared with traditional image systems. However, only high-resolution multispectral (HrMS) and low-resolution hyperspectral (LrHS) images can generally be captured at video rate in practice. In this paper, we propose a model-based deep learning approach for merging an HrMS and LrHS images to generate a high-resolution hyperspectral (HrHS) image. In specific, we construct a novel MS/HS fusion model which takes the observation models of low-resolution images and the low-rankness knowledge along the spectral mode of HrHS image into consideration. Then we design an iterative algorithm to solve the model by exploiting the proximal gradient method. And then, by unfolding the designed algorithm, we construct a deep network, called MS/HS Fusion Net, with learning the proximal operators and model parameters by convolutional neural networks. Experimental results on simulated and real data substantiate the superiority of our method both visually and quantitatively as compared with state-of-the-art methods along this line of research.