MambaSeg: Harnessing Mamba for Accurate and Efficient Image-Event Semantic Segmentation

TL;DR

MambaSeg is a dual-branch framework that effectively fuses RGB and event data for accurate, efficient semantic segmentation, leveraging novel modules to improve cross-modal alignment and outperform existing methods.

Contribution

The paper introduces MambaSeg, a dual-encoder architecture with a new interaction module for efficient multimodal fusion of RGB and event streams in segmentation tasks.

Findings

Achieves state-of-the-art performance on DDD17 and DSEC datasets.

Reduces computational cost compared to existing multimodal methods.

Enhances cross-modal alignment through the Dual-Dimensional Interaction Module.

Abstract

Semantic segmentation is a fundamental task in computer vision with wide-ranging applications, including autonomous driving and robotics. While RGB-based methods have achieved strong performance with CNNs and Transformers, their effectiveness degrades under fast motion, low-light, or high dynamic range conditions due to limitations of frame cameras. Event cameras offer complementary advantages such as high temporal resolution and low latency, yet lack color and texture, making them insufficient on their own. To address this, recent research has explored multimodal fusion of RGB and event data; however, many existing approaches are computationally expensive and focus primarily on spatial fusion, neglecting the temporal dynamics inherent in event streams. In this work, we propose MambaSeg, a novel dual-branch semantic segmentation framework that employs parallel Mamba encoders to efficiently model RGB images and event streams. To reduce cross-modal ambiguity, we introduce the Dual-Dimensional Interaction Module (DDIM), comprising a Cross-Spatial Interaction Module (CSIM) and a Cross-Temporal Interaction Module (CTIM), which jointly perform fine-grained fusion along both spatial and temporal dimensions. This design improves cross-modal alignment, reduces ambiguity, and leverages the complementary properties of each modality. Extensive experiments on the DDD17 and DSEC datasets demonstrate that MambaSeg achieves state-of-the-art segmentation performance while significantly reducing computational cost, showcasing its promise for efficient, scalable, and robust multimodal perception.