CLIOPATRA: Extracting Private Information from LLM Insights

TL;DR

This paper demonstrates that current heuristic privacy protections in LLM insight systems like Clio are vulnerable to sophisticated attacks, which can successfully extract sensitive user information despite layered defenses.

Contribution

It introduces CLIOPATRA, the first privacy attack against privacy-preserving LLM insight systems, revealing significant vulnerabilities in existing heuristic privacy protections.

Findings

CLIOPATRA successfully extracts medical history in 39% of cases with minimal adversary knowledge.

The attack reaches nearly 100% success with enhanced adversary knowledge and advanced models.

Existing privacy mitigations like LLM-based auditing are unreliable and often fail to detect leaks.

Abstract

As AI assistants become widely used, privacy-aware platforms like Anthropic's Clio have been introduced to generate insights from real-world AI use. Clio's privacy protections rely on layering multiple heuristic techniques together, including PII redaction, clustering, filtering, and LLM-based privacy auditing. In this paper, we put these claims to the test by presenting CLIOPATRA, the first privacy attack against "privacy-preserving" LLM insight systems. The attack involves a realistic adversary that carefully designs and inserts malicious chats into the system to break multiple layers of privacy protections and induce the leakage of sensitive information from a target user's chat. We evaluated CLIOPATRA on synthetically generated medical target chats, demonstrating that an adversary who knows only the basic demographics of a target user and a single symptom can successfully extract the user's medical history in 39% of cases by just inspecting Clio's output. Furthermore, CLIOPATRA can reach close to 100% when Clio is configured with other state-of-the-art models and the adversary's knowledge of the target user is increased. We also show that existing ad hoc mitigations, such as LLM-based privacy auditing, are unreliable and fail to detect major leaks. Our findings indicate that even when layered, current heuristic protections are insufficient to adequately protect user data in LLM-based analysis systems.