Med-MMFL: A Multimodal Federated Learning Benchmark in Healthcare

TL;DR

Med-MMFL introduces the first comprehensive multimodal federated learning benchmark for healthcare, evaluating diverse modalities, tasks, and algorithms to facilitate standardized assessment and future research in medical MMFL.

Contribution

This work provides the first standardized, multi-modality federated learning benchmark in healthcare, covering various datasets, tasks, and algorithms to advance systematic evaluation.

Findings

Evaluated six state-of-the-art FL algorithms across multiple modalities and tasks.

Demonstrated the benchmark's utility in assessing model performance in realistic heterogeneity settings.

Provided open-source implementation for reproducibility and future research.

Abstract

Federated learning (FL) enables collaborative model training across decentralized medical institutions while preserving data privacy. However, medical FL benchmarks remain scarce, with existing efforts focusing mainly on unimodal or bimodal modalities and a limited range of medical tasks. This gap underscores the need for standardized evaluation to advance systematic understanding in medical MultiModal FL (MMFL). To this end, we introduce Med-MMFL, the first comprehensive MMFL benchmark for the medical domain, encompassing diverse modalities, tasks, and federation scenarios. Our benchmark evaluates six representative state-of-the-art FL algorithms, covering different aggregation strategies, loss formulations, and regularization techniques. It spans datasets with 2 to 4 modalities, comprising a total of 10 unique medical modalities, including text, pathology images, ECG, X-ray, radiology reports, and multiple MRI sequences. Experiments are conducted across naturally federated, synthetic IID, and synthetic non-IID settings to simulate real-world heterogeneity. We assess segmentation, classification, modality alignment (retrieval), and VQA tasks. To support reproducibility and fair comparison of future multimodal federated learning (MMFL) methods under realistic medical settings, we release the complete benchmark implementation, including data processing and partitioning pipelines, at https://github.com/bhattarailab/Med-MMFL-Benchmark .