Evaluating LLM Adaptation to Sociodemographic Factors: User Profile vs. Dialogue History

TL;DR

This paper evaluates how well large language models adapt their responses based on sociodemographic user profiles and dialogue history, revealing that stronger reasoning models show better alignment with user attributes.

Contribution

It introduces a framework for comparing explicit user profiles and implicit dialogue history in assessing LLM sociodemographic adaptation, highlighting the importance of reasoning capabilities.

Findings

Models adjust their expressed values based on demographic changes.

Consistency of adaptation varies across models.

Stronger reasoning models show better alignment with user attributes.

Abstract

Effective engagement by large language models (LLMs) requires adapting responses to users' sociodemographic characteristics, such as age, occupation, and education level. While many real-world applications leverage dialogue history for contextualization, existing evaluations of LLMs' behavioral adaptation often focus on single-turn prompts. In this paper, we propose a framework to evaluate LLM adaptation when attributes are introduced either (1) explicitly via user profiles in the prompt or (2) implicitly through multi-turn dialogue history. We assess the consistency of model behavior across these modalities. Using a multi-agent pipeline, we construct a synthetic dataset pairing dialogue histories with distinct user profiles and employ questions from the Value Survey Module (VSM 2013) (Hofstede and Hofstede, 2016) to probe value expression. Our findings indicate that most models adjust their expressed values in response to demographic changes, particularly in age and education level, but consistency varies. Models with stronger reasoning capabilities demonstrate greater alignment, indicating the importance of reasoning in robust sociodemographic adaptation.