Personalized Federated Learning via Dual-Prompt Optimization and Cross Fusion

TL;DR

This paper introduces pFedDC, a personalized federated learning framework that uses dual prompts and cross fusion to handle data heterogeneity, improving model personalization across diverse datasets.

Contribution

The paper proposes a novel dual-prompt and cross-fusion approach for personalized federated learning, addressing limitations of existing text-only prompt methods and capturing client-specific data characteristics.

Findings

pFedDC outperforms state-of-the-art methods across nine datasets.

The dual-prompt design effectively captures shared and client-specific knowledge.

Cross fusion enhances personalized representation generation.

Abstract

Federated learning (FL) enables collaborative model training across decentralized clients without sharing local data, but is challenged by heterogeneity in data, computation, and communication. Pretrained vision-language models (VLMs), with their strong generalization and lightweight tuning via prompts, offer a promising solution. However, existing federated prompt-learning methods rely only on text prompts and overlook joint label-domain distribution shifts. In this paper, we propose a personalized FL framework based on dual-prompt learning and cross fusion, termed pFedDC. Specifically, each client maintains both global and local prompts across vision and language modalities: global prompts capture common knowledge shared across the federation, while local prompts encode client-specific semantics and domain characteristics. Meanwhile, a cross-fusion module is designed to adaptively integrate prompts from different levels, enabling the model to generate personalized representations aligned with each client's unique data distribution. Extensive experiments across nine datasets with various types of heterogeneity show that pFedDC consistently outperforms state-of-the-art methods.