CFMW: Cross-modality Fusion Mamba for Robust Object Detection under Adverse Weather

TL;DR

This paper introduces CFMW, a robust and efficient cross-modality fusion method for object detection in adverse weather, utilizing a new dataset and achieving state-of-the-art results.

Contribution

The paper presents CFMW, a novel fusion framework with weather-removal capabilities and a new severe weather dataset, improving robustness and efficiency in challenging conditions.

Findings

CFMW is 3 times faster than Transformer-based fusion methods.

CFMW achieves state-of-the-art detection performance on multiple datasets.

The SWVI dataset contains over 64,000 diverse weather-affected image pairs.

Abstract

Visible-infrared image pairs provide complementary information, enhancing the reliability and robustness of object detection applications in real-world scenarios. However, most existing methods face challenges in maintaining robustness under complex weather conditions, which limits their applicability. Meanwhile, the reliance on attention mechanisms in modality fusion introduces significant computational complexity and storage overhead, particularly when dealing with high-resolution images. To address these challenges, we propose the Cross-modality Fusion Mamba with Weather-removal (CFMW) to augment stability and cost-effectiveness under adverse weather conditions. Leveraging the proposed Perturbation-Adaptive Diffusion Model (PADM) and Cross-modality Fusion Mamba (CFM) modules, CFMW is able to reconstruct visual features affected by adverse weather, enriching the representation of image details. With efficient architecture design, CFMW is 3 times faster than Transformer-style fusion (e.g., CFT). To bridge the gap in relevant datasets, we construct a new Severe Weather Visible-Infrared (SWVI) dataset, encompassing diverse adverse weather scenarios such as rain, haze, and snow. The dataset contains 64,281 paired visible-infrared images, providing a valuable resource for future research. Extensive experiments on public datasets (i.e., M3FD and LLVIP) and the newly constructed SWVI dataset conclusively demonstrate that CFMW achieves state-of-the-art detection performance. Both the dataset and source code will be made publicly available at https://github.com/lhy-zjut/CFMW.