MoCTEFuse: Illumination-Gated Mixture of Chiral Transformer Experts for Multi-Level Infrared and Visible Image Fusion

TL;DR

MoCTEFuse is a novel multi-level image fusion network that adaptively balances infrared and visible image details by using illumination-gated experts and a dynamic transformer-based fusion mechanism, improving fusion quality and detection accuracy.

Contribution

The paper introduces MoCTEFuse, a dynamic fusion network with illumination-gated experts and a novel loss function, advancing infrared-visible image fusion by effectively handling illumination variations.

Findings

Outperforms existing methods on multiple datasets.

Achieves state-of-the-art detection mAP on MFNet and DroneVehicle.

Demonstrates robustness across diverse illumination conditions.

Abstract

While illumination changes inevitably affect the quality of infrared and visible image fusion, many outstanding methods still ignore this factor and directly merge the information from source images, leading to modality bias in the fused results. To this end, we propose a dynamic multi-level image fusion network called MoCTEFuse, which applies an illumination-gated Mixture of Chiral Transformer Experts (MoCTE) to adaptively preserve texture details and object contrasts in balance. MoCTE consists of high- and low-illumination expert subnetworks, each built upon the Chiral Transformer Fusion Block (CTFB). Guided by the illumination gating signals, CTFB dynamically switches between the primary and auxiliary modalities as well as assigning them corresponding weights with its asymmetric cross-attention mechanism. Meanwhile, it is stacked at multiple stages to progressively aggregate and refine modality-specific and cross-modality information. To facilitate robust training, we propose a competitive loss function that integrates illumination distributions with three levels of sub-loss terms. Extensive experiments conducted on the DroneVehicle, MSRS, TNO and RoadScene datasets show MoCTEFuse's superior fusion performance. Finally, it achieves the best detection mean Average Precision (mAP) of 70.93% on the MFNet dataset and 45.14% on the DroneVehicle dataset. The code and model are released at https://github.com/Bitlijinfu/MoCTEFuse.