Multi-modal Learning with Missing Modality via Shared-Specific Feature Modelling

TL;DR

The paper introduces ShaSpec, a novel multi-modal learning method that effectively handles missing modalities by learning shared and specific features, improving performance across various tasks like classification and segmentation.

Contribution

ShaSpec is a simple, effective approach that leverages auxiliary tasks and residual fusion to handle missing modalities and adapt to multiple tasks.

Findings

Outperforms existing methods significantly in medical image segmentation and classification.

Achieves over 3% improvement on BraTS2018 for tumor segmentation.

Easily adaptable to different multi-modal tasks.

Abstract

The missing modality issue is critical but non-trivial to be solved by multi-modal models. Current methods aiming to handle the missing modality problem in multi-modal tasks, either deal with missing modalities only during evaluation or train separate models to handle specific missing modality settings. In addition, these models are designed for specific tasks, so for example, classification models are not easily adapted to segmentation tasks and vice versa. In this paper, we propose the Shared-Specific Feature Modelling (ShaSpec) method that is considerably simpler and more effective than competing approaches that address the issues above. ShaSpec is designed to take advantage of all available input modalities during training and evaluation by learning shared and specific features to better represent the input data. This is achieved from a strategy that relies on auxiliary tasks based on distribution alignment and domain classification, in addition to a residual feature fusion procedure. Also, the design simplicity of ShaSpec enables its easy adaptation to multiple tasks, such as classification and segmentation. Experiments are conducted on both medical image segmentation and computer vision classification, with results indicating that ShaSpec outperforms competing methods by a large margin. For instance, on BraTS2018, ShaSpec improves the SOTA by more than 3% for enhancing tumour, 5% for tumour core and 3% for whole tumour. The code repository address is https://github.com/billhhh/ShaSpec/.