ICAFusion: Iterative Cross-Attention Guided Feature Fusion for Multispectral Object Detection

TL;DR

ICAFusion introduces a dual cross-attention transformer framework with iterative interaction to improve multispectral object detection, addressing misalignment issues and reducing model complexity for better performance.

Contribution

The paper proposes a novel dual cross-attention transformer framework with iterative parameter sharing for multispectral object detection, enhancing feature fusion and reducing complexity.

Findings

Achieves superior detection performance on KAIST, FLIR, and VEDAI datasets.

Provides faster inference compared to existing methods.

Demonstrates robustness to image misalignment.

Abstract

Effective feature fusion of multispectral images plays a crucial role in multi-spectral object detection. Previous studies have demonstrated the effectiveness of feature fusion using convolutional neural networks, but these methods are sensitive to image misalignment due to the inherent deffciency in local-range feature interaction resulting in the performance degradation. To address this issue, a novel feature fusion framework of dual cross-attention transformers is proposed to model global feature interaction and capture complementary information across modalities simultaneously. This framework enhances the discriminability of object features through the query-guided cross-attention mechanism, leading to improved performance. However, stacking multiple transformer blocks for feature enhancement incurs a large number of parameters and high spatial complexity. To handle this, inspired by the human process of reviewing knowledge, an iterative interaction mechanism is proposed to share parameters among block-wise multimodal transformers, reducing model complexity and computation cost. The proposed method is general and effective to be integrated into different detection frameworks and used with different backbones. Experimental results on KAIST, FLIR, and VEDAI datasets show that the proposed method achieves superior performance and faster inference, making it suitable for various practical scenarios. Code will be available at https://github.com/chanchanchan97/ICAFusion.