PrivacyBench: A Conversational Benchmark for Evaluating Privacy in Personalized AI

TL;DR

PrivacyBench is a new benchmark designed to evaluate how well conversational AI systems preserve user secrets, revealing significant privacy leaks in current models and highlighting the need for privacy-by-design safeguards.

Contribution

Introduces PrivacyBench, a novel conversational benchmark with socially grounded datasets to measure secret preservation and evaluate privacy risks in AI assistants.

Findings

Retrieval-Augmented Generation assistants leak secrets in up to 26.56% of interactions.

Privacy-aware prompts reduce leakage to 5.12%.

Current architectures rely heavily on generator safeguards, creating single points of failure.

Abstract

Personalized AI agents rely on access to a user's digital footprint, which often includes sensitive data from private emails, chats and purchase histories. Yet this access creates a fundamental societal and privacy risk: systems lacking social-context awareness can unintentionally expose user secrets, threatening digital well-being. We introduce PrivacyBench, a benchmark with socially grounded datasets containing embedded secrets and a multi-turn conversational evaluation to measure secret preservation. Testing Retrieval-Augmented Generation (RAG) assistants reveals that they leak secrets in up to 26.56% of interactions. A privacy-aware prompt lowers leakage to 5.12%, yet this measure offers only partial mitigation. The retrieval mechanism continues to access sensitive data indiscriminately, which shifts the entire burden of privacy preservation onto the generator. This creates a single point of failure, rendering current architectures unsafe for wide-scale deployment. Our findings underscore the urgent need for structural, privacy-by-design safeguards to ensure an ethical and inclusive web for everyone.