Continuous Spectral Reconstruction from RGB Images via Implicit Neural Representation

TL;DR

This paper introduces Neural Spectral Reconstruction (NeSR), a novel method using implicit neural representations to achieve continuous spectral reconstruction from RGB images, improving accuracy and flexibility over traditional discrete methods.

Contribution

NeSR is the first approach to model spectral signatures as a continuous function, allowing arbitrary spectral band outputs and better capturing spectral continuity.

Findings

NeSR outperforms baseline methods in spectral reconstruction accuracy.

NeSR enables flexible spectral band selection as an arbitrary output.

NeSR effectively models continuous spectral signatures from RGB images.

Abstract

Existing methods for spectral reconstruction usually learn a discrete mapping from RGB images to a number of spectral bands. However, this modeling strategy ignores the continuous nature of spectral signature. In this paper, we propose Neural Spectral Reconstruction (NeSR) to lift this limitation, by introducing a novel continuous spectral representation. To this end, we embrace the concept of implicit function and implement a parameterized embodiment with a neural network. Specifically, we first adopt a backbone network to extract spatial features of RGB inputs. Based on it, we devise Spectral Profile Interpolation (SPI) module and Neural Attention Mapping (NAM) module to enrich deep features, where the spatial-spectral correlation is involved for a better representation. Then, we view the number of sampled spectral bands as the coordinate of continuous implicit function, so as to learn the projection from deep features to spectral intensities. Extensive experiments demonstrate the distinct advantage of NeSR in reconstruction accuracy over baseline methods. Moreover, NeSR extends the flexibility of spectral reconstruction by enabling an arbitrary number of spectral bands as the target output.