Hyperspectral Image Super-Resolution with Spectral Mixup and Heterogeneous Datasets

TL;DR

This paper introduces a multi-task learning approach with Spectral Mixup augmentation and semi-supervised training for hyperspectral image super-resolution, effectively leveraging heterogeneous datasets and improving performance over existing methods.

Contribution

It proposes a novel multi-task network correlating HSI SR with RGB SR, introduces Spectral Mixup for data augmentation, and extends to semi-supervised learning from low-resolution datasets.

Findings

Outperforms existing methods on four datasets

Effectively leverages heterogeneous datasets

Improves super-resolution quality significantly

Abstract

This work studies Hyperspectral image (HSI) super-resolution (SR). HSI SR is characterized by high-dimensional data and a limited amount of training examples. This exacerbates the undesirable behaviors of neural networks such as memorization and sensitivity to out-of-distribution samples. This work addresses these issues with three contributions. First, we observe that HSI SR and RGB image SR are correlated and develop a novel multi-tasking network to train them jointly so that the auxiliary task RGB image SR can provide additional supervision. Second, we propose a simple, yet effective data augmentation routine, termed Spectral Mixup, to construct effective virtual training samples to enlarge the training set. Finally, we extend the network to a semi-supervised setting so that it can learn from datasets containing only low-resolution HSIs. With these contributions, our method is able to learn from heterogeneous datasets and lift the requirement for having a large amount of HD HSI training samples. Extensive experiments on four standard datasets show that our method outperforms existing methods significantly and underpin the relevance of our contributions. Code has been made available at https://github.com/kli8996/HSISR.