DAIQ: Auditing Demographic Attribute Inference from Question in LLMs

TL;DR

This paper introduces DAIQ, a framework for diagnosing whether large language models infer sensitive demographic attributes from neutral questions, revealing widespread inference behaviors and proposing abstention prompts to mitigate privacy concerns.

Contribution

The paper presents DAIQ, a novel diagnostic method for assessing demographic inference in LLMs, highlighting pervasive inference from neutral prompts and proposing mitigation strategies.

Findings

Many models infer demographics from neutral questions.

Models tend to default to socially dominant categories.

Abstention prompts significantly reduce unintended inference.

Abstract

Recent evaluations of Large language models (LLMs) audit social bias primarily through prompts that explicitly reference demographic attributes, overlooking whether models infer sensitive demographics from neutral questions. Such inference constitutes epistemic overreach and raises concerns for privacy. We introduce Demographic Attribute Inference from Questions (DAIQ), a diagnostic audit framework for evaluating demographic inference under epistemic uncertainty. We evaluate 18 open- and closed-source LLMs across six real-world domains and five demographic attributes. We find that many models infer demographics from neutral questions, defaulting to socially dominant categories and producing stereotype-aligned rationales. These behaviors persist across model families, scales and decoding settings, indicating reliance on learned population priors. We further show that inferred demographics can condition downstream responses and that abstention oriented prompting substantially reduces unintended inference without model fine-tuning. Our results suggest that current bias evaluations are incomplete and motivate evaluation standards that assess not only how models respond to demographic information, but whether they should infer it at all.