Learning Robust Anymodal Segmentor with Unimodal and Cross-modal Distillation

TL;DR

This paper introduces a novel framework for training robust multimodal segmentors that effectively handle any combination of visual modalities by using cross-modal and unimodal distillation techniques, improving performance and reducing modality bias.

Contribution

The paper presents the first comprehensive framework for unimodal and cross-modal distillation in multimodal segmentation, enhancing robustness across various modality combinations.

Findings

Achieves superior performance on synthetic and real-world benchmarks.

Effectively reduces unimodal bias and over-reliance on specific modalities.

Demonstrates robustness in handling missing or incomplete modalities.

Abstract

Simultaneously using multimodal inputs from multiple sensors to train segmentors is intuitively advantageous but practically challenging. A key challenge is unimodal bias, where multimodal segmentors over rely on certain modalities, causing performance drops when others are missing, common in real world applications. To this end, we develop the first framework for learning robust segmentor that can handle any combinations of visual modalities. Specifically, we first introduce a parallel multimodal learning strategy for learning a strong teacher. The cross-modal and unimodal distillation is then achieved in the multi scale representation space by transferring the feature level knowledge from multimodal to anymodal segmentors, aiming at addressing the unimodal bias and avoiding over-reliance on specific modalities. Moreover, a prediction level modality agnostic semantic distillation is proposed to achieve semantic knowledge transferring for segmentation. Extensive experiments on both synthetic and real-world multi-sensor benchmarks demonstrate that our method achieves superior performance.