Learned Spectral Super-Resolution

TL;DR

This paper introduces a CNN-based method for blind spectral super-resolution, generating high-resolution hyperspectral images from RGB inputs by learning spectral statistics, outperforming existing techniques on natural and satellite images.

Contribution

It is the first to leverage learned spectral statistics for blind single-image spectral super-resolution using deep learning.

Findings

Outperforms state-of-the-art methods on natural images

Effective for both RGB and satellite multi-spectral data

Demonstrates the feasibility of spectral enhancement from single images

Abstract

We describe a novel method for blind, single-image spectral super-resolution. While conventional super-resolution aims to increase the spatial resolution of an input image, our goal is to spectrally enhance the input, i.e., generate an image with the same spatial resolution, but a greatly increased number of narrow (hyper-spectral) wave-length bands. Just like the spatial statistics of natural images has rich structure, which one can exploit as prior to predict high-frequency content from a low resolution image, the same is also true in the spectral domain: the materials and lighting conditions of the observed world induce structure in the spectrum of wavelengths observed at a given pixel. Surprisingly, very little work exists that attempts to use this diagnosis and achieve blind spectral super-resolution from single images. We start from the conjecture that, just like in the spatial domain, we can learn the statistics of natural image spectra, and with its help generate finely resolved hyper-spectral images from RGB input. Technically, we follow the current best practice and implement a convolutional neural network (CNN), which is trained to carry out the end-to-end mapping from an entire RGB image to the corresponding hyperspectral image of equal size. We demonstrate spectral super-resolution both for conventional RGB images and for multi-spectral satellite data, outperforming the state-of-the-art.