What Do LLMs Associate with Your Name? A Human-Centered Black-Box Audit of Personal Data

TL;DR

This paper introduces a human-centered audit tool to evaluate how large language models associate personal data with individuals, revealing models' capabilities and user concerns about privacy and control.

Contribution

It presents LMP2, a privacy-preserving audit method, and provides empirical insights into LLMs' associations with personal data and user perceptions.

Findings

Models confidently generate personal data for well-known individuals.

GPT-4o accurately predicts multiple personal data features for everyday users.

Majority of users want control over model-generated personal data associations.

Abstract

Large language models (LLMs), and conversational agents based on them, are exposed to personal data (PD) during pre-training and during user interactions. Prior work shows that PD can resurface, yet users lack insight into how strongly models associate specific information to their identity. We audit PD across eight LLMs (3 open-source; 5 API-based, including GPT-4o), introduce LMP2 (Language Model Privacy Probe), a human-centered, privacy-preserving audit tool refined through two formative studies (N=20), and run two studies with EU residents to capture (i) intuitions about LLM-generated PD (N1=155) and (ii) reactions to tool output (N2=303). We show empirically that models confidently generate multiple PD categories for well-known individuals. For everyday users, GPT-4o generates 11 features with 60% or more accuracy (e.g., gender, hair color, languages). Finally, 72% of participants sought control over model-generated associations with their name, raising questions about what counts as PD and whether data privacy rights should extend to LLMs.