FreDFT: Frequency Domain Fusion Transformer for Visible-Infrared Object Detection

TL;DR

FreDFT introduces a frequency domain transformer for improved visible-infrared object detection, effectively addressing information imbalance and enhancing cross-modal feature fusion in complex scenarios.

Contribution

It proposes a novel frequency domain fusion transformer with multimodal attention, cross-modal global modeling, and local feature enhancement modules for superior detection performance.

Findings

Achieves state-of-the-art results on multiple datasets.

Effectively mitigates information imbalance in multimodal data.

Enhances feature fusion through frequency domain attention.

Abstract

Visible-infrared object detection has gained sufficient attention due to its detection performance in low light, fog, and rain conditions. However, visible and infrared modalities captured by different sensors exist the information imbalance problem in complex scenarios, which can cause inadequate cross-modal fusion, resulting in degraded detection performance. \textcolor{red}{Furthermore, most existing methods use transformers in the spatial domain to capture complementary features, ignoring the advantages of developing frequency domain transformers to mine complementary information.} To solve these weaknesses, we propose a frequency domain fusion transformer, called FreDFT, for visible-infrared object detection. The proposed approach employs a novel multimodal frequency domain attention (MFDA) to mine complementary information between modalities and a frequency domain feed-forward layer (FDFFL) via a mixed-scale frequency feature fusion strategy is designed to better enhance multimodal features. To eliminate the imbalance of multimodal information, a cross-modal global modeling module (CGMM) is constructed to perform pixel-wise inter-modal feature interaction in a spatial and channel manner. Moreover, a local feature enhancement module (LFEM) is developed to strengthen multimodal local feature representation and promote multimodal feature fusion by using various convolution layers and applying a channel shuffle. Extensive experimental results have verified that our proposed FreDFT achieves excellent performance on multiple public datasets compared with other state-of-the-art methods. The code of our FreDFT is linked at https://github.com/WenCongWu/FreDFT.