PrivaCI-Bench: Evaluating Privacy with Contextual Integrity and Legal Compliance

TL;DR

PrivaCI-Bench is a comprehensive benchmark for evaluating privacy in large language models based on Contextual Integrity theory, focusing on legal compliance and social context rather than just PII, revealing current models' limitations.

Contribution

This work introduces PrivaCI-Bench, a novel privacy evaluation benchmark that incorporates social context and legal regulations to assess LLMs' privacy and safety compliance.

Findings

LLMs can capture key CI parameters within a context

Current LLMs need improvements for full privacy compliance

Benchmark covers real court cases, policies, and synthetic data

Abstract

Recent advancements in generative large language models (LLMs) have enabled wider applicability, accessibility, and flexibility. However, their reliability and trustworthiness are still in doubt, especially for concerns regarding individuals' data privacy. Great efforts have been made on privacy by building various evaluation benchmarks to study LLMs' privacy awareness and robustness from their generated outputs to their hidden representations. Unfortunately, most of these works adopt a narrow formulation of privacy and only investigate personally identifiable information (PII). In this paper, we follow the merit of the Contextual Integrity (CI) theory, which posits that privacy evaluation should not only cover the transmitted attributes but also encompass the whole relevant social context through private information flows. We present PrivaCI-Bench, a comprehensive contextual privacy evaluation benchmark targeted at legal compliance to cover well-annotated privacy and safety regulations, real court cases, privacy policies, and synthetic data built from the official toolkit to study LLMs' privacy and safety compliance. We evaluate the latest LLMs, including the recent reasoner models QwQ-32B and Deepseek R1. Our experimental results suggest that though LLMs can effectively capture key CI parameters inside a given context, they still require further advancements for privacy compliance.