FedDEAP: Adaptive Dual-Prompt Tuning for Multi-Domain Federated Learning

TL;DR

FedDEAP introduces an adaptive prompt tuning framework that enhances CLIP's ability to generalize across multiple domains in federated learning by disentangling semantic and domain features and balancing shared and personalized prompts.

Contribution

The paper proposes a novel federated prompt tuning method, FedDEAP, that improves multi-domain CLIP performance by disentangling features and using dual prompts for better generalization.

Findings

Significant improvement in multi-domain image recognition accuracy.

Effective preservation of domain-specific knowledge during federated training.

Theoretical analysis confirms the robustness of the proposed method.

Abstract

Federated learning (FL) enables multiple clients to collaboratively train machine learning models without exposing local data, balancing performance and privacy. However, domain shift and label heterogeneity across clients often hinder the generalization of the aggregated global model. Recently, large-scale vision-language models like CLIP have shown strong zero-shot classification capabilities, raising the question of how to effectively fine-tune CLIP across domains in a federated setting. In this work, we propose an adaptive federated prompt tuning framework, FedDEAP, to enhance CLIP's generalization in multi-domain scenarios. Our method includes the following three key components: (1) To mitigate the loss of domain-specific information caused by label-supervised tuning, we disentangle semantic and domain-specific features in images by using semantic and domain transformation networks with unbiased mappings; (2) To preserve domain-specific knowledge during global prompt aggregation, we introduce a dual-prompt design with a global semantic prompt and a local domain prompt to balance shared and personalized information; (3) To maximize the inclusion of semantic and domain information from images in the generated text features, we align textual and visual representations under the two learned transformations to preserve semantic and domain consistency. Theoretical analysis and extensive experiments on four datasets demonstrate the effectiveness of our method in enhancing the generalization of CLIP for federated image recognition across multiple domains.