A Joint Convolution Auto-encoder Network for Infrared and Visible Image Fusion

TL;DR

This paper introduces a joint convolution auto-encoder network that effectively fuses infrared and visible images by separately learning redundant and complementary features, resulting in high-quality fused images.

Contribution

The proposed JCAE network uniquely combines private and common branches to learn and fuse redundant and complementary features simultaneously for infrared and visible image fusion.

Findings

Achieves superior subjective visual quality

Outperforms existing methods on objective metrics

Effectively separates and fuses features from infrared and visible images

Abstract

Background: Leaning redundant and complementary relationships is a critical step in the human visual system. Inspired by the infrared cognition ability of crotalinae animals, we design a joint convolution auto-encoder (JCAE) network for infrared and visible image fusion. Methods: Our key insight is to feed infrared and visible pair images into the network simultaneously and separate an encoder stream into two private branches and one common branch, the private branch works for complementary features learning and the common branch does for redundant features learning. We also build two fusion rules to integrate redundant and complementary features into their fused feature which are then fed into the decoder layer to produce the final fused image. We detail the structure, fusion rule and explain its multi-task loss function. Results: Our JCAE network achieves good results in terms of both subjective effect and objective evaluation metrics.