SecP-Tuning: Efficient Privacy-Preserving Prompt Tuning for Large Language Models via MPC

TL;DR

SecP-Tuning introduces an efficient MPC-based framework for privacy-preserving prompt tuning of large language models, significantly reducing computational and communication costs while maintaining competitive performance.

Contribution

It is the first to enable efficient privacy-preserving prompt tuning of LLMs using MPC, with innovative forward-only tuning and a novel privacy-preserving self-attention mechanism.

Findings

Achieves 12x faster end-to-end training compared to full fine-tuning.

Reduces communication overhead by up to 20x.

Maintains comparable task performance to gradient-based methods.

Abstract

Large Language Models (LLMs) have revolutionized numerous fields, yet their adaptation to specialized tasks in privacy-sensitive domains such as healthcare and finance remains constrained due to the scarcity of accessible training data caused by stringent privacy requirements. Secure Multi-party Computation (MPC)-based privacy-preserving machine learning provides theoretical guarantees for the privacy of model parameters and data. However, its application to LLMs has been predominantly limited to inference, as fine-tuning introduces significant efficiency challenges, particularly in backward propagation, optimizer, and self-attention operations. To address these challenges, we propose SecP-Tuning, the first MPC-based framework designed for efficient, privacy-preserving prompt tuning of LLMs. SecP-Tuning innovatively integrates Forward-only Tuning (FoT) through the ``data owner-server interaction" paradigm, effectively removing the need for privacy-preserving computations in backward propagation and optimization processes. Furthermore, it devises an efficient privacy-preserving Random Feature Attention (RFA), effectively mitigating the computational complexity of softmax-based self-attention and circumventing MPC-incompatible nonlinear operations. Experimental results demonstrate that, compared to full-Parameter Supervised Fine-Tuning (SFT) and gradient-based prompt tuning, SecP-Tuning achieves approximately 12x and 16x end-to-end acceleration, as well as 17x and 20x reductions in communication overhead, respectively. Moreover, it delivers performance comparable to gradient-based methods across multiple few-shot tasks.