Learning deep illumination-robust features from multispectral filter array images

TL;DR

This paper introduces a novel method for learning illumination-robust, discriminant features directly from raw multispectral images, improving classification accuracy and reducing computational costs compared to existing approaches.

Contribution

It proposes raw spectral constancy, MSFA-preserving transformations, and raw-mixing techniques to enhance deep learning from raw multispectral images under varying lighting conditions.

Findings

Outperforms existing methods in MS image classification

Requires less computational effort

Enhances robustness to illumination variations

Abstract

Multispectral (MS) snapshot cameras equipped with a MS filter array (MSFA), capture multiple spectral bands in a single shot, resulting in a raw mosaic image where each pixel holds only one channel value. The fully-defined MS image is estimated from the raw one through \textit{demosaicing}, which inevitably introduces spatio-spectral artifacts. Moreover, training on fully-defined MS images can be computationally intensive, particularly with deep neural networks (DNNs), and may result in features lacking discrimination power due to suboptimal learning of spatio-spectral interactions. Furthermore, outdoor MS image acquisition occurs under varying lighting conditions, leading to illumination-dependent features. This paper presents an original approach to learn discriminant and illumination-robust features directly from raw images. It involves: \textit{raw spectral constancy} to mitigate the impact of illumination, \textit{MSFA-preserving} transformations suited for raw image augmentation to train DNNs on diverse raw textures, and \textit{raw-mixing} to capture discriminant spatio-spectral interactions in raw images. Experiments on MS image classification show that our approach outperforms both handcrafted and recent deep learning-based methods, while also requiring significantly less computational effort. The source code is available at https://github.com/AnisAmziane/RawTexture.