DMM: Disparity-guided Multispectral Mamba for Oriented Object Detection in Remote Sensing

TL;DR

This paper introduces DMM, an efficient multispectral object detection framework that uses disparity-guided fusion and attention modules to improve accuracy while reducing computational complexity.

Contribution

The paper proposes DMM, a novel multispectral detection method combining disparity-guided fusion, multi-scale attention, and target-aware auxiliary tasks for better performance.

Findings

Outperforms state-of-the-art methods on DroneVehicle and VEDAI datasets.

Achieves high detection accuracy with lower computational cost.

Effectively mitigates inter-modal and intra-modal discrepancies.

Abstract

Multispectral oriented object detection faces challenges due to both inter-modal and intra-modal discrepancies. Recent studies often rely on transformer-based models to address these issues and achieve cross-modal fusion detection. However, the quadratic computational complexity of transformers limits their performance. Inspired by the efficiency and lower complexity of Mamba in long sequence tasks, we propose Disparity-guided Multispectral Mamba (DMM), a multispectral oriented object detection framework comprised of a Disparity-guided Cross-modal Fusion Mamba (DCFM) module, a Multi-scale Target-aware Attention (MTA) module, and a Target-Prior Aware (TPA) auxiliary task. The DCFM module leverages disparity information between modalities to adaptively merge features from RGB and IR images, mitigating inter-modal conflicts. The MTA module aims to enhance feature representation by focusing on relevant target regions within the RGB modality, addressing intra-modal variations. The TPA auxiliary task utilizes single-modal labels to guide the optimization of the MTA module, ensuring it focuses on targets and their local context. Extensive experiments on the DroneVehicle and VEDAI datasets demonstrate the effectiveness of our method, which outperforms state-of-the-art methods while maintaining computational efficiency. Code will be available at https://github.com/Another-0/DMM.