M2I2HA: Multi-modal Object Detection Based on Intra- and Inter-Modal Hypergraph Attention

TL;DR

This paper introduces M2I2HA, a hypergraph-based multi-modal detection network that effectively models complex relationships within and across modalities, achieving state-of-the-art results in challenging environments.

Contribution

The paper proposes a novel hypergraph-based architecture with intra- and inter-modal modules for enhanced multi-modal feature extraction and fusion in object detection.

Findings

Achieves state-of-the-art detection accuracy on multiple datasets.

Effectively models high-order relationships within each modality.

Enhances cross-modal feature alignment and fusion.

Abstract

Recent advances in multi-modal detection have significantly improved detection accuracy in challenging environments (e.g., low light, overexposure). By integrating RGB with modalities such as thermal and depth, multi-modal fusion increases data redundancy and system robustness. However, significant challenges remain in effectively extracting task-relevant information both within and across modalities, as well as in achieving precise cross-modal alignment. While CNNs excel at feature extraction, they are limited by constrained receptive fields, strong inductive biases, and difficulty in capturing long-range dependencies. Transformer-based models offer global context but suffer from quadratic computational complexity and are confined to pairwise correlation modeling. Mamba and other State Space Models (SSMs), on the other hand, are hindered by their sequential scanning mechanism, which flattens 2D spatial structures into 1D sequences, disrupting topological relationships and limiting the modeling of complex higher-order dependencies. To address these issues, we propose a multi-modal perception network based on hypergraph theory called M2I2HA. Our architecture includes an Intra-Hypergraph Enhancement module to capture global many-to-many high-order relationships within each modality, and an Inter-Hypergraph Fusion module to align, enhance, and fuse cross-modal features by bridging configuration and spatial gaps between data sources. We further introduce a M2-FullPAD module to enable adaptive multi-level fusion of multi-modal enhanced features within the network, meanwhile enhancing data distribution and flow across the architecture. Extensive object detection experiments on multiple public datasets against baselines demonstrate that M2I2HA achieves state-of-the-art performance in multi-modal object detection tasks.