An Analysis of Large Language Models for Simulating User Responses in Surveys

TL;DR

This paper evaluates the ability of Large Language Models to simulate diverse human survey responses, highlighting their limitations in representing varied viewpoints and reasoning over demographic nuances.

Contribution

It introduces CLAIMSIM, a method to diversify LLM responses, and provides an analysis of LLM biases and reasoning limitations in survey simulation.

Findings

CLAIMSIM increases response diversity.

LLMs tend to maintain fixed viewpoints regardless of demographics.

LLMs struggle with nuanced reasoning over conflicting claims.

Abstract

Using Large Language Models (LLMs) to simulate user opinions has received growing attention. Yet LLMs, especially trained with reinforcement learning from human feedback (RLHF), are known to exhibit biases toward dominant viewpoints, raising concerns about their ability to represent users from diverse demographic and cultural backgrounds. In this work, we examine the extent to which LLMs can simulate human responses to cross-domain survey questions through direct prompting and chain-of-thought prompting. We further propose a claim diversification method CLAIMSIM, which elicits viewpoints from LLM parametric knowledge as contextual input. Experiments on the survey question answering task indicate that, while CLAIMSIM produces more diverse responses, both approaches struggle to accurately simulate users. Further analysis reveals two key limitations: (1) LLMs tend to maintain fixed viewpoints across varying demographic features, and generate single-perspective claims; and (2) when presented with conflicting claims, LLMs struggle to reason over nuanced differences among demographic features, limiting their ability to adapt responses to specific user profiles.