Are Large Language Models Chameleons? An Attempt to Simulate Social Surveys

TL;DR

This paper investigates the potential of large language models to simulate social survey responses, revealing significant biases and variability influenced by prompts, and proposing new metrics for comparison with real survey data.

Contribution

It introduces a novel similarity measure inspired by Jaccard similarity and emphasizes the importance of analyzing prompt robustness for accurate social survey simulation using LLMs.

Findings

LLMs exhibit cultural, age, and gender biases in responses.

Prompt choice significantly affects bias and variability.

A new similarity metric helps compare LLM responses with survey data.

Abstract

Can large language models (LLMs) simulate social surveys? To answer this question, we conducted millions of simulations in which LLMs were asked to answer subjective questions. A comparison of different LLM responses with the European Social Survey (ESS) data suggests that the effect of prompts on bias and variability is fundamental, highlighting major cultural, age, and gender biases. We further discussed statistical methods for measuring the difference between LLM answers and survey data and proposed a novel measure inspired by Jaccard similarity, as LLM-generated responses are likely to have a smaller variance. Our experiments also reveal that it is important to analyze the robustness and variability of prompts before using LLMs to simulate social surveys, as their imitation abilities are approximate at best.