Differentially Private In-context Learning via Sampling Few-shot Mixed with Zero-shot Outputs

TL;DR

This paper introduces a novel decoding framework, dps-mozo, that enhances privacy in in-context learning for generation tasks by sampling from mixed outputs, achieving differential privacy without noise and maintaining high utility.

Contribution

The paper proposes dps-mozo, a sampling-based method for differentially private in-context learning that reduces information leakage without utility loss.

Findings

Achieves strong privacy guarantee with ε=2

Minimal utility degradation with only 0.3% ROUGE-L F1 decrease

Effective privacy-utility tradeoff in open-ended generation tasks

Abstract

In-context learning (ICL) has shown promising improvement in downstream task adaptation of LLMs by augmenting prompts with relevant input-output examples (demonstrations). However, the ICL demonstrations can contain privacy-sensitive information, which can be leaked and/or regurgitated by the LLM output. Differential Privacy (DP), a widely adopted privacy safeguard, has emerged to mitigate this privacy leakage, with recent work demonstrating strong privacy-utility tradeoffs in classification tasks for ICL. However, generation tasks for ICL are challenging due to the high-dimensional output space of open-ended generation. To this end, we propose $\texttt{dps-mozo}$, Differentially Private Sampling by Mixing One-shot with Zero-shot Outputs, a decoding framework that generates DP text by sampling from the product of multiple one-shot outputs mixed with a zero-shot output. This mixing effectively reduces the amount of information that can be leaked by each demonstration. By utilizing the inherent randomness in sampling from the mixed distributions, we can achieve DP without adding noise, thereby improving the privacy-utility tradeoff. Our experimental evaluations show $\texttt{dps-mozo}$ can achieve a strong privacy guarantee, $\epsilon=2$, with minimal utility degradation compared to non-private few-shot learning, $\textbf{0.3}$% ROUGE-L F1 score decrease on the SAMSum dataset with Gemma 2 2B.