Efficient Federated Prompt Tuning for Black-box Large Pre-trained Models

TL;DR

This paper introduces Fed-BBPT, a federated prompt tuning method that enables efficient, privacy-preserving adaptation of large pre-trained models using black-box API access, without fine-tuning model parameters.

Contribution

The paper proposes a novel federated black-box prompt tuning approach that avoids parameter fine-tuning and preserves data privacy, suitable for large models accessed via APIs.

Findings

Effective across 40 datasets in CV and NLP.

Reduces memory and privacy issues compared to traditional fine-tuning.

Achieves competitive performance without model parameter access.

Abstract

With the blowout development of pre-trained models (PTMs), the efficient tuning of these models for diverse downstream applications has emerged as a pivotal research concern. Although recent investigations into prompt tuning have provided promising avenues, three salient challenges persist: (1) memory constraint: the continuous growth in the size of open-source PTMs renders fine-tuning, even a fraction of their parameters, challenging for many practitioners. (2) model privacy: existing PTMs often function as public API services, with their parameters inaccessible for effective or tailored fine-tuning. (3) data privacy: the fine-tuning of PTMs necessitates high-quality datasets, which are typically localized and not shared to public. To optimally harness each local dataset while navigating memory constraints and preserving privacy, we propose Federated Black-Box Prompt Tuning (Fed-BBPT). This innovative approach eschews reliance on parameter architectures and private dataset access, instead capitalizing on a central server that aids local users in collaboratively training a prompt generator through regular aggregation. Local users leverage API-driven learning via a zero-order optimizer, obviating the need for PTM deployment. Relative to extensive fine-tuning, Fed-BBPT proficiently sidesteps memory challenges tied to PTM storage and fine-tuning on local machines, tapping into comprehensive, high-quality, yet private training datasets. A thorough evaluation across 40 datasets spanning CV and NLP tasks underscores the robustness of our proposed model.