Model-Guided Network with Cluster-Based Operators for Spatio-Spectral Super-Resolution

TL;DR

This paper introduces a novel end-to-end model-driven framework for joint spatio-spectral super-resolution of hyperspectral images, combining variational methods with learnable modules for improved reconstruction quality.

Contribution

It proposes a decomposed approach with tailored learnable operators for spatial, spectral, and fusion tasks, integrating classical algorithms with deep learning techniques.

Findings

Effective in reconstructing high-resolution hyperspectral images

Outperforms existing methods on multiple datasets

Demonstrates robustness across different sampling factors

Abstract

This paper addresses the problem of reconstructing a high-resolution hyperspectral image from a low-resolution multispectral observation. While spatial super-resolution and spectral super-resolution have been extensively studied, joint spatio-spectral super-resolution remains relatively explored. We propose an end-to-end model-driven framework that explicitly decomposes the joint spatio-spectral super-resolution problem into spatial super-resolution, spectral super-resolution and fusion tasks. Each sub-task is addressed by unfolding a variational-based approach, where the operators involved in the proximal gradient iterative scheme are replaced with tailored learnable modules. In particular, we design an upsampling operator for spatial super-resolution based on classical back-projection algorithms, adapted to handle arbitrary scaling factors. Spectral reconstruction is performed using learnable cluster-based upsampling and downsampling operators. For image fusion, we integrate low-frequency estimation and high-frequency injection modules to combine the spatial and spectral information from spatial super-resolution and spectral super-resolution outputs. Additionally, we introduce an efficient nonlocal post-processing step that leverages image self-similarity by combining a multi-head attention mechanism with residual connections. Extensive evaluations on several datasets and sampling factors demonstrate the effectiveness of our approach. The source code will be available at https://github.com/TAMI-UIB/JSSUNet