From Demographics to Survey Anchors: Evaluating LLM Agents for Modeling Retirement Attitudes

TL;DR

This study evaluates the effectiveness of demographic-only versus survey-informed LLM agents in predicting retirement attitudes, revealing limitations of demographic-only models in capturing human response variability.

Contribution

It demonstrates that demographic-only LLM agents are less accurate and fail to replicate human response patterns compared to survey-anchored agents.

Findings

Demographic-only agents exhibit central tendency bias.

They are unrealistically accurate, missing 'don't know' responses.

Survey-anchored agents better reproduce complex interactions.

Abstract

Large language models (LLM) agents may offer tools to predict human responses to surveys. A common technique for defining these agents uses only demographics, for example country, age, gender, employment status, income, education and marital status. We compare the predictive accuracy of demographic agents to that of survey agents defined with a larger set of in-domain survey responses. We test both approaches in predicting responses to the multidisciplinary, cross-national Survey of Health, Ageing and Retirement in Europe (SHARE), focusing on five variables from three policy-relevant constructs around personal finance. In these three constructs, we observe that, compared to survey agents trained on broader data, demographics-only agents (1) exhibited a central tendency bias, skewing answers toward population means, and (2) were unrealistically accurate, failing to reproduce the incorrect answers and "don't know" responses typical of human respondents. These performance differences are further substantiated through the replication of a hierarchical regression analysis from prior retirement planning research. Agents based solely on demographic information reproduce the outcome that financial risk tolerance, future time perspective, and knowledge of retirement planning each are predictive of retirement savings. However, only the survey-anchored agents succeed in reproducing the interaction among these three factors. These findings suggest caution in using only demographics to define LLM agents for predicting survey responses.