EGFormer: Towards Efficient and Generalizable Multimodal Semantic Segmentation

TL;DR

EGFormer is an efficient multimodal semantic segmentation framework that dynamically prioritizes and filters modalities, reducing computational costs significantly while maintaining high performance and demonstrating strong generalization in transfer tasks.

Contribution

The paper introduces EGFormer, a novel framework with modules for dynamic modality importance scoring and dropping, enabling efficient and generalizable multimodal segmentation.

Findings

Achieves up to 88% reduction in parameters

Reduces GFLOPs by 50% while maintaining accuracy

Outperforms existing methods in transfer learning tasks

Abstract

Recent efforts have explored multimodal semantic segmentation using various backbone architectures. However, while most methods aim to improve accuracy, their computational efficiency remains underexplored. To address this, we propose EGFormer, an efficient multimodal semantic segmentation framework that flexibly integrates an arbitrary number of modalities while significantly reducing model parameters and inference time without sacrificing performance. Our framework introduces two novel modules. First, the Any-modal Scoring Module (ASM) assigns importance scores to each modality independently, enabling dynamic ranking based on their feature maps. Second, the Modal Dropping Module (MDM) filters out less informative modalities at each stage, selectively preserving and aggregating only the most valuable features. This design allows the model to leverage useful information from all available modalities while discarding redundancy, thus ensuring high segmentation quality. In addition to efficiency, we evaluate EGFormer on a synthetic-to-real transfer task to demonstrate its generalizability. Extensive experiments show that EGFormer achieves competitive performance with up to 88 percent reduction in parameters and 50 percent fewer GFLOPs. Under unsupervised domain adaptation settings, it further achieves state-of-the-art transfer performance compared to existing methods.