Multispectral Texture Synthesis using RGB Convolutional Neural Networks

TL;DR

This paper extends RGB texture synthesis algorithms to multispectral images using two methods that do not require retraining neural networks, enabling effective exemplar-based multispectral texture synthesis with good visual quality.

Contribution

It introduces two novel approaches for multispectral texture synthesis that adapt existing RGB methods without additional network training.

Findings

Methods achieve comparable quality to RGB state-of-the-art techniques.

Projection-based approach benefits from upstream color transfer.

Approaches perform well across various metrics.

Abstract

State-of-the-art RGB texture synthesis algorithms rely on style distances that are computed through statistics of deep features. These deep features are extracted by classification neural networks that have been trained on large datasets of RGB images. Extending such synthesis methods to multispectral images is not straightforward, since the pre-trained networks are designed for and have been trained on RGB images. In this work, we propose two solutions to extend these methods to multispectral imaging. Neither of them require additional training of the neural network from which the second order neural statistics are extracted. The first one consists in optimizing over batches of random triplets of spectral bands throughout training. The second one projects multispectral pixels onto a 3 dimensional space. We further explore the benefit of a color transfer operation upstream of the projection to avoid the potentially abnormal color distributions induced by the projection. Our experiments compare the performances of the various methods through different metrics. We demonstrate that they can be used to perform exemplar-based texture synthesis, achieve good visual quality and comes close to state-of-the art methods on RGB bands.