Multispectral State-Space Feature Fusion: Bridging Shared and Cross-Parametric Interactions for Object Detection

TL;DR

This paper introduces MS2Fusion, a novel multispectral feature fusion framework based on state space models that effectively combines shared semantics and cross-modal interactions, improving object detection and perception tasks.

Contribution

The paper proposes a dual-path state-space model framework for multispectral feature fusion, enhancing generalization and scalability over existing methods.

Findings

MS2Fusion outperforms state-of-the-art multispectral object detection methods on benchmarks.

It achieves state-of-the-art results on RGB-T semantic segmentation.

The framework is applicable to various multispectral perception tasks.

Abstract

Modern multispectral feature fusion for object detection faces two critical limitations: (1) Excessive preference for local complementary features over cross-modal shared semantics adversely affects generalization performance; and (2) The trade-off between the receptive field size and computational complexity present critical bottlenecks for scalable feature modeling. Addressing these issues, a novel Multispectral State-Space Feature Fusion framework, dubbed MS2Fusion, is proposed based on the state space model (SSM), achieving efficient and effective fusion through a dual-path parametric interaction mechanism. More specifically, the first cross-parameter interaction branch inherits the advantage of cross-attention in mining complementary information with cross-modal hidden state decoding in SSM. The second shared-parameter branch explores cross-modal alignment with joint embedding to obtain cross-modal similar semantic features and structures through parameter sharing in SSM. Finally, these two paths are jointly optimized with SSM for fusing multispectral features in a unified framework, allowing our MS2Fusion to enjoy both functional complementarity and shared semantic space. In our extensive experiments on mainstream benchmarks including FLIR, M3FD and LLVIP, our MS2Fusion significantly outperforms other state-of-the-art multispectral object detection methods, evidencing its superiority. Moreover, MS2Fusion is general and applicable to other multispectral perception tasks. We show that, even without specific design, MS2Fusion achieves state-of-the-art results on RGB-T semantic segmentation and RGBT salient object detection, showing its generality. The source code will be available at https://github.com/61s61min/MS2Fusion.git.