Differentially Private and Communication Efficient Large Language Model Split Inference via Stochastic Quantization and Soft Prompt

TL;DR

This paper introduces DEL, a novel framework that enhances privacy and reduces communication costs in large language model inference by combining differential privacy, stochastic quantization, and soft prompts, eliminating the need for local models.

Contribution

The work is the first to leverage soft prompts to balance privacy and utility in privacy-preserving LLM split inference, integrating differential privacy with communication efficiency.

Findings

Effective privacy-utility trade-off demonstrated on benchmarks.

Significant reduction in communication overhead.

Maintains high performance in text generation and understanding tasks.

Abstract

Large Language Models (LLMs) have achieved remarkable performance and received significant research interest. The enormous computational demands, however, hinder the local deployment on devices with limited resources. The current prevalent LLM inference paradigms require users to send queries to the service providers for processing, which raises critical privacy concerns. Existing approaches propose to allow the users to obfuscate the token embeddings before transmission and utilize local models for denoising. Nonetheless, transmitting the token embeddings and deploying local models may result in excessive communication and computation overhead, preventing practical implementation. In this work, we propose \textbf{DEL}, a framework for \textbf{D}ifferentially private and communication \textbf{E}fficient \textbf{L}LM split inference. More specifically, an embedding projection module and a differentially private stochastic quantization mechanism are proposed to reduce the communication overhead in a privacy-preserving manner. To eliminate the need for local models, we adapt soft prompt at the server side to compensate for the utility degradation caused by privacy. To the best of our knowledge, this is the first work that utilizes soft prompt to improve the trade-off between privacy and utility in LLM inference, and extensive experiments on text generation and natural language understanding benchmarks demonstrate the effectiveness of the proposed method.