FedPrompt: Communication-Efficient and Privacy Preserving Prompt Tuning in Federated Learning

TL;DR

FedPrompt introduces a communication-efficient, privacy-preserving prompt tuning method for federated learning that significantly reduces communication costs and demonstrates robustness against backdoor attacks in NLP tasks.

Contribution

The paper proposes FedPrompt, a novel prompt tuning approach in federated learning that reduces communication costs and enhances security against backdoor threats.

Findings

Split aggregation reduces communication to 0.01% of PLMs' parameters.

FedPrompt maintains high accuracy on IID and Non-IID data.

Normal backdoor attacks have low success rates against FedPrompt.

Abstract

Federated learning (FL) has enabled global model training on decentralized data in a privacy-preserving way by aggregating model updates. However, for many natural language processing (NLP) tasks that utilize pre-trained language models (PLMs) with large numbers of parameters, there are considerable communication costs associated with FL. Recently, prompt tuning, which tunes some soft prompts without modifying PLMs, has achieved excellent performance as a new learning paradigm. Therefore we want to combine the two methods and explore the effect of prompt tuning under FL. In this paper, we propose "FedPrompt" to study prompt tuning in a model split aggregation way using FL, and prove that split aggregation greatly reduces the communication cost, only 0.01% of the PLMs' parameters, with little decrease on accuracy both on IID and Non-IID data distribution. This improves the efficiency of FL method while also protecting the data privacy in prompt tuning. In addition, like PLMs, prompts are uploaded and downloaded between public platforms and personal users, so we try to figure out whether there is still a backdoor threat using only soft prompts in FL scenarios. We further conduct backdoor attacks by data poisoning on FedPrompt. Our experiments show that normal backdoor attack can not achieve a high attack success rate, proving the robustness of FedPrompt. We hope this work can promote the application of prompt in FL and raise the awareness of the possible security threats.