FedMM: Federated Multi-Modal Learning with Modality Heterogeneity in Computational Pathology

TL;DR

FedMM introduces a federated learning framework that trains separate modality-specific feature extractors in computational pathology, improving privacy and performance despite modality heterogeneity across hospitals.

Contribution

It proposes a novel federated multi-modal learning approach that trains separate feature extractors, addressing modality heterogeneity and privacy concerns in computational pathology.

Findings

FedMM outperforms baseline methods in accuracy.

FedMM achieves higher AUC metrics.

Effective even with small datasets or limited devices.

Abstract

The fusion of complementary multimodal information is crucial in computational pathology for accurate diagnostics. However, existing multimodal learning approaches necessitate access to users' raw data, posing substantial privacy risks. While Federated Learning (FL) serves as a privacy-preserving alternative, it falls short in addressing the challenges posed by heterogeneous (yet possibly overlapped) modalities data across various hospitals. To bridge this gap, we propose a Federated Multi-Modal (FedMM) learning framework that federatedly trains multiple single-modal feature extractors to enhance subsequent classification performance instead of existing FL that aims to train a unified multimodal fusion model. Any participating hospital, even with small-scale datasets or limited devices, can leverage these federated trained extractors to perform local downstream tasks (e.g., classification) while ensuring data privacy. Through comprehensive evaluations of two publicly available datasets, we demonstrate that FedMM notably outperforms two baselines in accuracy and AUC metrics.