SegRGB-X: General RGB-X Semantic Segmentation Model

TL;DR

SegRGB-X introduces a universal framework for RGB-X semantic segmentation, unifying multiple modalities with innovative modules, achieving state-of-the-art results across diverse datasets.

Contribution

The paper presents a novel universal segmentation framework with modality-aware guidance, aligned embeddings, and dynamic feature refinement, enabling effective multi-modal segmentation.

Findings

Achieves a mIoU of 65.03% on diverse datasets.

Outperforms specialized multi-modal methods.

Demonstrates effectiveness across five different sensor modalities.

Abstract

Semantic segmentation across arbitrary sensor modalities faces significant challenges due to diverse sensor characteristics, and the traditional configurations for this task result in redundant development efforts. We address these challenges by introducing a universal arbitrary-modal semantic segmentation framework that unifies segmentation across multiple modalities. Our approach features three key innovations: (1) the Modality-aware CLIP (MA-CLIP), which provides modality-specific scene understanding guidance through LoRA fine-tuning; (2) Modality-aligned Embeddings for capturing fine-grained features; and (3) the Domain-specific Refinement Module (DSRM) for dynamic feature adjustment. Evaluated on five diverse datasets with different complementary modalities (event, thermal, depth, polarization, and light field), our model surpasses specialized multi-modal methods and achieves state-of-the-art performance with a mIoU of 65.03%. The codes will be released upon acceptance.