Can Large Language Models Really Recognize Your Name?

TL;DR

This paper demonstrates that large language models often fail to reliably recognize human names in ambiguous contexts, raising concerns about privacy and fairness in LLM-based data protection systems.

Contribution

The authors introduce AmBench, a benchmark of ambiguous human names, and reveal significant limitations in LLMs' ability to recognize these names, impacting privacy enforcement.

Findings

LLMs' recall for ambiguous names drops by 20-40%

Contextual ambiguity causes LLMs to mishandle broad classes of names

Benign prompt injections can significantly reduce name recognition in enterprise tools

Abstract

Large language models (LLMs) are increasingly being used in privacy pipelines to detect and remedy sensitive data leakage. These solutions often rely on the premise that LLMs can reliably recognize human names, one of the most important categories of personally identifiable information (PII). In this paper, we reveal how LLMs can consistently mishandle broad classes of human names even in short text snippets due to ambiguous linguistic cues in the contexts. We construct AmBench, a benchmark of over 12,000 real yet ambiguous human names based on the name regularity bias phenomenon. Each name appears in dozens of concise text snippets that are compatible with multiple entity types. Our experiments with 12 state-of-the-art LLMs show that the recall of AmBench names drops by 20--40% compared to more recognizable names. This uneven privacy protection due to linguistic properties raises important concerns about the fairness of privacy enforcement. When the contexts contain benign prompt injections -- instruction-like user texts that can cause LLMs to conflate data with commands -- AmBench names can become four times more likely to be ignored in Clio, an LLM-powered enterprise tool used by Anthropic AI to extract supposedly privacy-preserving insights from user conversations with Claude. Our findings showcase blind spots in the performance and fairness of LLM-based privacy solutions and call for a systematic investigation into their privacy failure modes and countermeasures.