Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generation

TL;DR

This paper introduces pFedPG, a federated learning framework that personalizes large-scale models efficiently by generating client-specific prompts, improving adaptation to heterogeneous data while reducing computation and communication costs.

Contribution

It proposes a novel prompt generation approach for personalized federated learning that leverages large pre-trained models without fine-tuning entire networks.

Findings

Outperforms state-of-the-art personalized FL methods on benchmark datasets.

Reduces computation and communication costs compared to traditional fine-tuning.

Effectively adapts large models to diverse client data distributions.

Abstract

Federated learning (FL) emerges as a decentralized learning framework which trains models from multiple distributed clients without sharing their data to preserve privacy. Recently, large-scale pre-trained models (e.g., Vision Transformer) have shown a strong capability of deriving robust representations. However, the data heterogeneity among clients, the limited computation resources, and the communication bandwidth restrict the deployment of large-scale models in FL frameworks. To leverage robust representations from large-scale models while enabling efficient model personalization for heterogeneous clients, we propose a novel personalized FL framework of client-specific Prompt Generation (pFedPG), which learns to deploy a personalized prompt generator at the server for producing client-specific visual prompts that efficiently adapts frozen backbones to local data distributions. Our proposed framework jointly optimizes the stages of personalized prompt adaptation locally and personalized prompt generation globally. The former aims to train visual prompts that adapt foundation models to each client, while the latter observes local optimization directions to generate personalized prompts for all clients. Through extensive experiments on benchmark datasets, we show that our pFedPG is favorable against state-of-the-art personalized FL methods under various types of data heterogeneity, allowing computation and communication efficient model personalization.