Multi-Modal Style Transfer-based Prompt Tuning for Efficient Federated Domain Generalization

TL;DR

This paper introduces FaST-PT, a federated domain generalization framework utilizing multi-modal style transfer and prompt decomposition to improve unseen domain adaptation efficiently across distributed clients.

Contribution

It proposes a novel lightweight style transfer method and a dual-prompt module with domain-aware prompt generation for better federated domain generalization.

Findings

FaST-PT outperforms state-of-the-art FDG methods on benchmark datasets.

The style transfer expands training data and reduces domain shift.

The dual-prompt approach enhances unseen domain adaptation.

Abstract

Federated Domain Generalization (FDG) aims to collaboratively train a global model across distributed clients that can generalize well on unseen domains. However, existing FDG methods typically struggle with cross-client data heterogeneity and incur significant communication and computation overhead. To address these challenges, this paper presents a new FDG framework, dubbed FaST-PT, which facilitates local feature augmentation and efficient unseen domain adaptation in a distributed manner. First, we propose a lightweight Multi-Modal Style Transfer (MST) method to transform image embedding under text supervision, which could expand the training data distribution and mitigate domain shift. We then design a dual-prompt module that decomposes the prompt into global and domain prompts. Specifically, global prompts capture general knowledge from augmented embedding across clients, while domain prompts capture domain-specific knowledge from local data. Besides, Domain-aware Prompt Generation (DPG) is introduced to adaptively generate suitable prompts for each sample, which facilitates unseen domain adaptation through knowledge fusion. Extensive experiments on four cross-domain benchmark datasets, e.g., PACS and DomainNet, demonstrate the superior performance of FaST-PT over SOTA FDG methods such as FedDG-GA and DiPrompt. Ablation studies further validate the effectiveness and efficiency of FaST-PT.