Examining Modality Incongruity in Multimodal Federated Learning for Medical Vision and Language-based Disease Detection

TL;DR

This paper investigates how missing modalities affect multimodal federated learning in medical imaging and language tasks, exploring solutions like attention mechanisms, modality imputation, and regularization to improve model robustness.

Contribution

It is the first to analyze the impact of modality incongruity in MMFL and evaluates methods to mitigate its effects in medical vision and language-based disease detection.

Findings

Incongruity impacts model performance significantly.

Self-attention mechanisms help fuse information effectively.

Modality imputation reduces missing modality issues.

Abstract

Multimodal Federated Learning (MMFL) utilizes multiple modalities in each client to build a more powerful Federated Learning (FL) model than its unimodal counterpart. However, the impact of missing modality in different clients, also called modality incongruity, has been greatly overlooked. This paper, for the first time, analyses the impact of modality incongruity and reveals its connection with data heterogeneity across participating clients. We particularly inspect whether incongruent MMFL with unimodal and multimodal clients is more beneficial than unimodal FL. Furthermore, we examine three potential routes of addressing this issue. Firstly, we study the effectiveness of various self-attention mechanisms towards incongruity-agnostic information fusion in MMFL. Secondly, we introduce a modality imputation network (MIN) pre-trained in a multimodal client for modality translation in unimodal clients and investigate its potential towards mitigating the missing modality problem. Thirdly, we assess the capability of client-level and server-level regularization techniques towards mitigating modality incongruity effects. Experiments are conducted under several MMFL settings on two publicly available real-world datasets, MIMIC-CXR and Open-I, with Chest X-Ray and radiology reports.