Learning A 3D-CNN and Transformer Prior for Hyperspectral Image Super-Resolution

TL;DR

This paper introduces a novel hyperspectral image super-resolution method combining Transformer-based priors with 3D-CNNs, effectively capturing spatial and spectral features for improved image quality.

Contribution

It proposes a new HSISR approach using Transformer and 3D-CNN to better model spatial and spectral dependencies, surpassing existing CNN-based methods.

Findings

Achieves significant performance improvements over mainstream algorithms.

Demonstrates effectiveness on multiple datasets and real-world data.

Outperforms both conventional and recent deep learning methods.

Abstract

To solve the ill-posed problem of hyperspectral image super-resolution (HSISR), an usually method is to use the prior information of the hyperspectral images (HSIs) as a regularization term to constrain the objective function. Model-based methods using hand-crafted priors cannot fully characterize the properties of HSIs. Learning-based methods usually use a convolutional neural network (CNN) to learn the implicit priors of HSIs. However, the learning ability of CNN is limited, it only considers the spatial characteristics of the HSIs and ignores the spectral characteristics, and convolution is not effective for long-range dependency modeling. There is still a lot of room for improvement. In this paper, we propose a novel HSISR method that uses Transformer instead of CNN to learn the prior of HSIs. Specifically, we first use the proximal gradient algorithm to solve the HSISR model, and then use an unfolding network to simulate the iterative solution processes. The self-attention layer of Transformer makes it have the ability of spatial global interaction. In addition, we add 3D-CNN behind the Transformer layers to better explore the spatio-spectral correlation of HSIs. Both quantitative and visual results on two widely used HSI datasets and the real-world dataset demonstrate that the proposed method achieves a considerable gain compared to all the mainstream algorithms including the most competitive conventional methods and the recently proposed deep learning-based methods.