Cross Spectral Image Reconstruction Using a Deep Guided Neural Network

TL;DR

This paper introduces DGNet, a deep neural network designed for reconstructing cross spectral images in multispectral camera arrays, outperforming existing methods in quality and speed, especially with limited training data.

Contribution

The paper presents a novel deep guided neural network with specialized regularization and data augmentation for cross spectral image reconstruction from limited data.

Findings

Outperforms state-of-the-art by up to 2 dB PSNR

Achieves faster runtime by nearly 12 times

Produces more visually appealing results on real data

Abstract

Cross spectral camera arrays, where each camera records different spectral content, are becoming increasingly popular for RGB, multispectral and hyperspectral imaging, since they are capable of a high resolution in every dimension using off-the-shelf hardware. For these, it is necessary to build an image processing pipeline to calculate a consistent image data cube, i.e., it should look like as if every camera records the scene from the center camera. Since the cameras record the scene from a different angle, this pipeline needs a reconstruction component for pixels that are not visible to peripheral cameras. For that, a novel deep guided neural network (DGNet) is presented. Since only little cross spectral data is available for training, this neural network is highly regularized. Furthermore, a new data augmentation process is introduced to generate the cross spectral content. On synthetic and real multispectral camera array data, the proposed network outperforms the state of the art by up to 2 dB in terms of PSNR on average. Besides, DGNet also tops its best competitor in terms of SSIM as well as in runtime by a factor of nearly 12. Moreover, a qualitative evaluation reveals visually more appealing results for real camera array data.