Federated CLIP for Resource-Efficient Heterogeneous Medical Image Classification

TL;DR

This paper introduces FedMedCLIP, a federated learning approach using CLIP for resource-efficient, privacy-preserving medical image classification across heterogeneous datasets, achieving high accuracy with reduced communication and computation costs.

Contribution

The paper proposes a novel federated CLIP-based framework with a masked feature adaptation module, private local classifiers, and adaptive distillation, addressing data heterogeneity and resource constraints in medical imaging.

Findings

Achieves 8% higher accuracy on ISIC2019 compared to second best baseline.

Provides 120× faster training than FedAVG.

Demonstrates effective performance across four medical datasets.

Abstract

Despite the remarkable performance of deep models in medical imaging, they still require source data for training, which limits their potential in light of privacy concerns. Federated learning (FL), as a decentralized learning framework that trains a shared model with multiple hospitals (a.k.a., FL clients), provides a feasible solution. However, data heterogeneity and resource costs hinder the deployment of FL models, especially when using vision language models (VLM). To address these challenges, we propose a novel contrastive language-image pre-training (CLIP) based FL approach for medical image classification (FedMedCLIP). Specifically, we introduce a masked feature adaptation module (FAM) as a communication module to reduce the communication load while freezing the CLIP encoders to reduce the computational overhead. Furthermore, we propose a masked multi-layer perceptron (MLP) as a private local classifier to adapt to the client tasks. Moreover, we design an adaptive Kullback-Leibler (KL) divergence-based distillation regularization method to enable mutual learning between FAM and MLP. Finally, we incorporate model compression to transmit the FAM parameters while using ensemble predictions for classification. Extensive experiments on four publicly available medical datasets demonstrate that our model provides feasible performance (e.g., 8\% higher compared to second best baseline on ISIC2019) with reasonable resource cost (e.g., 120$\times$ faster than FedAVG).