Different Demographic Cues Yield Inconsistent Conclusions About LLM Personalization and Bias

TL;DR

This study reveals that using different demographic cues like names or other signals leads to inconsistent conclusions about LLM personalization and bias, emphasizing the importance of multi-cue evaluation for reliable insights.

Contribution

It demonstrates that demographic cues are not interchangeable and that their differences significantly impact assessments of LLM responses, advocating for multi-cue, mechanism-aware evaluation methods.

Findings

Cues for the same demographic group produce only partially overlapping response changes.

Bias assessments vary in magnitude and direction depending on the cue used.

Inconsistencies are due to differences in cue-group association strength and linguistic features.

Abstract

Demographic cue-based evaluation is widely used to study how large language models (LLMs) adapt their responses to signaled demographic attributes within and across groups. This approach typically relies on a single cue (e.g., names) as a proxy for group membership, implicitly treating different cues as interchangeable operationalizations of the same identity-conditioned behavior. We test this assumption in realistic advice-seeking interactions spanning 14.8 million prompts, focusing on race and gender in a U.S. context. We find that cues for the same group induce only partially overlapping changes in model responses, yielding inconsistent conclusions about personalization, while bias conclusions are unstable, with both magnitude and direction of group differences varying across cues. We further show that these inconsistencies reflect differences in cue-group association strength and linguistic features bundled within cues that shape model responses. Together, our findings suggest that demographic conditioning in LLMs is not a cue-invariant category-level parameter but depends fundamentally on how identity is cued, reflecting responses to linguistic signals rather than stable demographic categories. We therefore advocate multi-cue, mechanism-aware evaluations for robust and interpretable claims about demographic variation in LLM responses.