Data-adaptive Differentially Private Prompt Synthesis for In-Context Learning

TL;DR

This paper introduces AdaDPSyn, a data-adaptive differentially private algorithm for generating synthetic data to enable private in-context learning with large language models, balancing privacy and accuracy.

Contribution

AdaDPSyn adaptively adjusts noise levels based on data properties, using a novel iterative radius reduction technique to improve privacy-preserving synthetic data generation for ICL.

Findings

Outperforms existing DP few-shot generation methods.

Maintains high ICL accuracy close to non-private baselines.

Provides formal differential privacy guarantees.

Abstract

Large Language Models (LLMs) rely on the contextual information embedded in examples/demonstrations to perform in-context learning (ICL). To mitigate the risk of LLMs potentially leaking private information contained in examples in the prompt, we introduce a novel data-adaptive differentially private algorithm called AdaDPSyn to generate synthetic examples from the private dataset and then use these synthetic examples to perform ICL. The objective of AdaDPSyn is to adaptively adjust the noise level in the data synthesis mechanism according to the inherent statistical properties of the data, thereby preserving high ICL accuracy while maintaining formal differential privacy guarantees. A key innovation in AdaDPSyn is the Precision-Focused Iterative Radius Reduction technique, which dynamically refines the aggregation radius - the scope of data grouping for noise addition - based on patterns observed in data clustering, thereby minimizing the amount of additive noise. We conduct extensive experiments on standard benchmarks and compare AdaDPSyn with DP few-shot generation algorithm (Tang et al., 2023). The experiments demonstrate that AdaDPSyn not only outperforms DP few-shot generation, but also maintains high accuracy levels close to those of non-private baselines, providing an effective solution for ICL with privacy protection.