Visible and infrared self-supervised fusion trained on a single example

TL;DR

This paper introduces a self-supervised neural network method for fusing visible and infrared images using only a single example, enabling quick, high-quality multispectral fusion without extensive training data.

Contribution

The novel approach trains a CNN with self-supervision on a single image pair, achieving effective multispectral fusion without heavy datasets or long training times.

Findings

Achieves comparable or superior fusion quality to state-of-the-art methods.

Requires only seconds of training per image pair.

Does not depend on large datasets or extensive training.

Abstract

Multispectral imaging is an important task of image processing and computer vision, which is especially relevant to applications such as dehazing or object detection. With the development of the RGBT (RGB & Thermal) sensor, the problem of visible (RGB) to Near Infrared (NIR) image fusion has become particularly timely. Indeed, while visible images see color, but suffer from noise, haze, and clouds, the NIR channel captures a clearer picture. The proposed approach fuses these two channels by training a Convolutional Neural Network by Self Supervised Learning (SSL) on a single example. For each such pair, RGB and NIR, the network is trained for seconds to deduce the final fusion. The SSL is based on the comparison of the Structure of Similarity and Edge-Preservation losses, where the labels for the SSL are the input channels themselves. This fusion preserves the relevant detail of each spectral channel without relying on a heavy training process. Experiments demonstrate that the proposed approach achieves similar or better qualitative and quantitative multispectral fusion results than other state-of-the-art methods that do not rely on heavy training and/or large datasets.