Differentially Private In-Context Learning with Nearest Neighbor Search

TL;DR

This paper introduces a differentially private in-context learning framework that incorporates privacy-aware nearest neighbor search, significantly improving privacy-utility trade-offs in large language model applications.

Contribution

We propose a novel DP framework for in-context learning that integrates privacy-preserving nearest neighbor retrieval, addressing a key component often overlooked in LLM pipelines.

Findings

Outperforms existing baselines across multiple benchmarks.

Achieves better privacy-utility trade-offs.

Demonstrates effectiveness on text classification and question answering.

Abstract

Differentially private in-context learning (DP-ICL) has recently become an active research topic due to the inherent privacy risks of in-context learning. However, existing approaches overlook a critical component of modern large language model (LLM) pipelines: the similarity search used to retrieve relevant context data. In this work, we introduce a DP framework for in-context learning that integrates nearest neighbor search of relevant examples in a privacy-aware manner. Our method outperforms existing baselines by a substantial margin across all evaluated benchmarks, achieving more favorable privacy-utility trade-offs. To achieve this, we employ nearest neighbor retrieval from a database of context data, combined with a privacy filter that tracks the cumulative privacy cost of selected samples to ensure adherence to a central differential privacy budget. Experimental results on text classification and document question answering show a clear advantage of the proposed method over existing baselines.