On the Privacy Risk of In-context Learning

TL;DR

This paper demonstrates that large language models pose significant privacy risks through membership inference attacks when used with prompts containing private data, and proposes ensembling as a mitigation strategy.

Contribution

It reveals the privacy vulnerabilities of in-context learning in LLMs and introduces ensembling to reduce membership inference risks.

Findings

Prompted models have higher privacy risks than fine-tuned models at similar utility levels.

Model confidence on prompted data correlates with increased privacy vulnerability.

Ensembling multiple prompted models reduces membership inference risk.

Abstract

Large language models (LLMs) are excellent few-shot learners. They can perform a wide variety of tasks purely based on natural language prompts provided to them. These prompts contain data of a specific downstream task -- often the private dataset of a party, e.g., a company that wants to leverage the LLM for their purposes. We show that deploying prompted models presents a significant privacy risk for the data used within the prompt by instantiating a highly effective membership inference attack. We also observe that the privacy risk of prompted models exceeds fine-tuned models at the same utility levels. After identifying the model's sensitivity to their prompts -- in the form of a significantly higher prediction confidence on the prompted data -- as a cause for the increased risk, we propose ensembling as a mitigation strategy. By aggregating over multiple different versions of a prompted model, membership inference risk can be decreased.