Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection

TL;DR

This paper introduces a novel target-aware dual adversarial learning framework for infrared and visible image fusion, enhancing object detection accuracy and visual quality, supported by a comprehensive multi-scenario benchmark.

Contribution

It proposes a bilevel optimization approach with a dual discriminator network that preserves target structures and textures, advancing fusion and detection performance.

Findings

Achieves higher detection mAP than state-of-the-art methods.

Produces visually appealing fused images with preserved structural details.

Validates effectiveness on multiple datasets and a new comprehensive benchmark.

Abstract

This study addresses the issue of fusing infrared and visible images that appear differently for object detection. Aiming at generating an image of high visual quality, previous approaches discover commons underlying the two modalities and fuse upon the common space either by iterative optimization or deep networks. These approaches neglect that modality differences implying the complementary information are extremely important for both fusion and subsequent detection task. This paper proposes a bilevel optimization formulation for the joint problem of fusion and detection, and then unrolls to a target-aware Dual Adversarial Learning (TarDAL) network for fusion and a commonly used detection network. The fusion network with one generator and dual discriminators seeks commons while learning from differences, which preserves structural information of targets from the infrared and textural details from the visible. Furthermore, we build a synchronized imaging system with calibrated infrared and optical sensors, and collect currently the most comprehensive benchmark covering a wide range of scenarios. Extensive experiments on several public datasets and our benchmark demonstrate that our method outputs not only visually appealing fusion but also higher detection mAP than the state-of-the-art approaches.