FACMIC: Federated Adaptative CLIP Model for Medical Image Classification

TL;DR

FACMIC introduces an efficient federated CLIP-based model with domain adaptation for improved medical image classification across decentralized datasets, reducing communication costs and enhancing performance.

Contribution

The paper proposes a lightweight, adaptive federated CLIP model with domain adaptation, addressing communication costs and data heterogeneity in medical image classification.

Findings

Superior performance on four datasets

Effective domain adaptation reduces data distribution differences

Efficient feature attention module improves client-specific feature selection

Abstract

Federated learning (FL) has emerged as a promising approach to medical image analysis that allows deep model training using decentralized data while ensuring data privacy. However, in the field of FL, communication cost plays a critical role in evaluating the performance of the model. Thus, transferring vision foundation models can be particularly challenging due to the significant resource costs involved. In this paper, we introduce a federated adaptive Contrastive Language Image Pretraining CLIP model designed for classification tasks. We employ a light-weight and efficient feature attention module for CLIP that selects suitable features for each client's data. Additionally, we propose a domain adaptation technique to reduce differences in data distribution between clients. Experimental results on four publicly available datasets demonstrate the superior performance of FACMIC in dealing with real-world and multisource medical imaging data. Our codes are available at https://github.com/AIPMLab/FACMIC.