Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning

TL;DR

This paper introduces SGPT, a novel prompt-based federated learning algorithm that effectively balances global and personalized model performance across heterogeneous data distributions using shared and group-specific prompts.

Contribution

SGPT uniquely combines shared and group-specific prompts with a prompt selection module and BCD training to improve federated learning under data heterogeneity.

Findings

SGPT outperforms state-of-the-art baselines in heterogeneous FL settings.

The prompt selection module enables automatic adaptation to diverse local data.

Theoretical analysis shows reduced gap between global and local performance.

Abstract

Vision Transformers (ViT) and Visual Prompt Tuning (VPT) achieve state-of-the-art performance with improved efficiency in various computer vision tasks. This suggests a promising paradigm shift of adapting pre-trained ViT models to Federated Learning (FL) settings. However, the challenge of data heterogeneity among FL clients presents a significant hurdle in effectively deploying ViT models. Existing Generalized FL (GFL) and Personalized FL (PFL) methods have limitations in balancing performance across both global and local data distributions. In this paper, we present a novel algorithm, SGPT, that integrates GFL and PFL approaches by employing a unique combination of both shared and group-specific prompts. This design enables SGPT to capture both common and group-specific features. A key feature of SGPT is its prompt selection module, which facilitates the training of a single global model capable of automatically adapting to diverse local client data distributions without the need for local fine-tuning. To effectively train the prompts, we utilize block coordinate descent (BCD), learning from common feature information (shared prompts), and then more specialized knowledge (group prompts) iteratively. Theoretically, we justify that learning the proposed prompts can reduce the gap between global and local performance. Empirically, we conduct experiments on both label and feature heterogeneity settings in comparison with state-of-the-art baselines, along with extensive ablation studies, to substantiate the superior performance of SGPT.