Position: Contextual Integrity is Inadequately Applied to Language Models

TL;DR

This paper critically examines how the concept of Contextual Integrity is currently misapplied to large language models, emphasizing the need for proper adherence to its core principles to ensure accurate privacy assessments.

Contribution

It clarifies the fundamental tenets of CI theory, analyzes prior work for deviations, and highlights overlooked issues in applying CI to LLMs.

Findings

Existing applications of CI to LLMs often neglect core principles.

Misapplication of CI can lead to flawed privacy conclusions.

Overlooked issues include prompt sensitivity and positional bias.

Abstract

Machine learning community is discovering Contextual Integrity (CI) as a useful framework to assess the privacy implications of large language models (LLMs). This is an encouraging development. The CI theory emphasizes sharing information in accordance with privacy norms and can bridge the social, legal, political, and technical aspects essential for evaluating privacy in LLMs. However, this is also a good point to reflect on use of CI for LLMs. This position paper argues that existing literature inadequately applies CI for LLMs without embracing the theory's fundamental tenets. Inadequate applications of CI could lead to incorrect conclusions and flawed privacy-preserving designs. We clarify the four fundamental tenets of CI theory, systematize prior work on whether they deviate from these tenets, and highlight overlooked issues in experimental hygiene for LLMs (e.g., prompt sensitivity, positional bias).