Estimating Privacy Leakage of Augmented Contextual Knowledge in Language Models

TL;DR

This paper introduces a new metric called context influence, based on differential privacy, to accurately estimate privacy leakage in language models caused by augmented contextual knowledge, addressing overestimation issues of previous methods.

Contribution

It proposes the context influence metric to measure privacy leakage from contextual knowledge in language models, effectively isolating the impact of augmented contexts.

Findings

Context privacy leakage occurs with out-of-distribution contexts.

Context influence accurately attributes privacy leakage to augmented contexts.

Factors like model size and context size affect privacy leakage.

Abstract

Language models (LMs) rely on their parametric knowledge augmented with relevant contextual knowledge for certain tasks, such as question answering. However, the contextual knowledge can contain private information that may be leaked when answering queries, and estimating this privacy leakage is not well understood. A straightforward approach of directly comparing an LM's output to the contexts can overestimate the privacy risk, since the LM's parametric knowledge might already contain the augmented contextual knowledge. To this end, we introduce *context influence*, a metric that builds on differential privacy, a widely-adopted privacy notion, to estimate the privacy leakage of contextual knowledge during decoding. Our approach effectively measures how each subset of the context influences an LM's response while separating the specific parametric knowledge of the LM. Using our context influence metric, we demonstrate that context privacy leakage occurs when contextual knowledge is out of distribution with respect to parametric knowledge. Moreover, we experimentally demonstrate how context influence properly attributes the privacy leakage to augmented contexts, and we evaluate how factors -- such as model size, context size, generation position, etc. -- affect context privacy leakage. The practical implications of our results will inform practitioners of the privacy risk associated with augmented contextual knowledge.