Customize Segment Anything Model for Multi-Modal Semantic Segmentation with Mixture of LoRA Experts

TL;DR

This paper adapts the Segment Anything Model for multi-modal semantic segmentation by introducing a Mixture of LoRA Experts, enabling effective multi-modal feature integration and significantly improving performance on various benchmarks.

Contribution

It proposes a novel MoE-LoRA adaptation for SAM, with a new routing strategy and multi-scale feature fusion, to enhance multi-modal segmentation performance.

Findings

Outperforms state-of-the-art methods on DELIVER, MUSES, and MCubeS benchmarks.

Achieves 32.15% performance gain in missing modality scenarios.

Effectively integrates multi-modal features with a new adaptive routing strategy.

Abstract

The recent Segment Anything Model (SAM) represents a significant breakthrough in scaling segmentation models, delivering strong performance across various downstream applications in the RGB modality. However, directly applying SAM to emerging visual modalities, such as depth and event data results in suboptimal performance in multi-modal segmentation tasks. In this paper, we make the first attempt to adapt SAM for multi-modal semantic segmentation by proposing a Mixture of Low-Rank Adaptation Experts (MoE-LoRA) tailored for different input visual modalities. By training only the MoE-LoRA layers while keeping SAM's weights frozen, SAM's strong generalization and segmentation capabilities can be preserved for downstream tasks. Specifically, to address cross-modal inconsistencies, we propose a novel MoE routing strategy that adaptively generates weighted features across modalities, enhancing multi-modal feature integration. Additionally, we incorporate multi-scale feature extraction and fusion by adapting SAM's segmentation head and introducing an auxiliary segmentation head to combine multi-scale features for improved segmentation performance effectively. Extensive experiments were conducted on three multi-modal benchmarks: DELIVER, MUSES, and MCubeS. The results consistently demonstrate that the proposed method significantly outperforms state-of-the-art approaches across diverse scenarios. Notably, under the particularly challenging condition of missing modalities, our approach exhibits a substantial performance gain, achieving an improvement of 32.15% compared to existing methods.