Social Simulations with Large Language Model Risk Utopian Illusion

TL;DR

This paper systematically analyzes how large language models simulate social behavior, revealing they produce overly idealized, biased representations that lack real human complexity, thus risking misinterpretation in social science applications.

Contribution

It introduces a novel framework for analyzing LLMs' social simulation behavior and uncovers their tendency to generate utopian, biased social interactions unlike genuine human behavior.

Findings

LLMs reflect social desirability bias in simulations

LLMs exhibit social role bias, primacy effect, positivity bias

Simulated societies lack real human complexity and variability

Abstract

Reliable simulation of human behavior is essential for explaining, predicting, and intervening in our society. Recent advances in large language models (LLMs) have shown promise in emulating human behaviors, interactions, and decision-making, offering a powerful new lens for social science studies. However, the extent to which LLMs diverge from authentic human behavior in social contexts remains underexplored, posing risks of misinterpretation in scientific studies and unintended consequences in real-world applications. Here, we introduce a systematic framework for analyzing LLMs' behavior in social simulation. Our approach simulates multi-agent interactions through chatroom-style conversations and analyzes them across five linguistic dimensions, providing a simple yet effective method to examine emergent social cognitive biases. We conduct extensive experiments involving eight representative LLMs across three families. Our findings reveal that LLMs do not faithfully reproduce genuine human behavior but instead reflect overly idealized versions of it, shaped by the social desirability bias. In particular, LLMs show social role bias, primacy effect, and positivity bias, resulting in "Utopian" societies that lack the complexity and variability of real human interactions. These findings call for more socially grounded LLMs that capture the diversity of human social behavior.