Multimodal Transformer Using Cross-Channel attention for Object Detection in Remote Sensing Images

TL;DR

This paper introduces a novel early-stage multi-modal fusion method using cross-channel attention and an enhanced SWIN transformer, significantly improving small object detection in remote sensing images.

Contribution

The paper proposes an early-stage multi-modal fusion strategy with cross-attention and an augmented SWIN transformer, addressing computational complexity and small object detection challenges.

Findings

Achieves superior detection performance compared to existing methods.

Effectively fuses multiple modalities at early stages for better accuracy.

Enhances SWIN transformer with convolution layers to improve local attention.

Abstract

Object detection in Remote Sensing Images (RSI) is a critical task for numerous applications in Earth Observation (EO). Differing from object detection in natural images, object detection in remote sensing images faces challenges of scarcity of annotated data and the presence of small objects represented by only a few pixels. Multi-modal fusion has been determined to enhance the accuracy by fusing data from multiple modalities such as RGB, infrared (IR), lidar, and synthetic aperture radar (SAR). To this end, the fusion of representations at the mid or late stage, produced by parallel subnetworks, is dominant, with the disadvantages of increasing computational complexity in the order of the number of modalities and the creation of additional engineering obstacles. Using the cross-attention mechanism, we propose a novel multi-modal fusion strategy for mapping relationships between different channels at the early stage, enabling the construction of a coherent input by aligning the different modalities. By addressing fusion in the early stage, as opposed to mid or late-stage methods, our method achieves competitive and even superior performance compared to existing techniques. Additionally, we enhance the SWIN transformer by integrating convolution layers into the feed-forward of non-shifting blocks. This augmentation strengthens the model's capacity to merge separated windows through local attention, thereby improving small object detection. Extensive experiments prove the effectiveness of the proposed multimodal fusion module and the architecture, demonstrating their applicability to object detection in multimodal aerial imagery.