Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs

TL;DR

This paper critically examines the limitations of using large language models for social interaction simulations, highlighting that they perform well under idealized omniscient conditions but struggle with realistic, asymmetric information scenarios.

Contribution

The authors develop an evaluation framework contrasting omniscient and non-omniscient social simulations with LLMs, revealing significant performance gaps in realistic settings.

Findings

LLMs excel in omniscient simulation environments.

LLMs struggle with information asymmetry in realistic scenarios.

Addressing information asymmetry is crucial for real-world social AI applications.

Abstract

Recent advances in large language models (LLM) have enabled richer social simulations, allowing for the study of various social phenomena. However, most recent work has used a more omniscient perspective on these simulations (e.g., single LLM to generate all interlocutors), which is fundamentally at odds with the non-omniscient, information asymmetric interactions that involve humans and AI agents in the real world. To examine these differences, we develop an evaluation framework to simulate social interactions with LLMs in various settings (omniscient, non-omniscient). Our experiments show that LLMs perform better in unrealistic, omniscient simulation settings but struggle in ones that more accurately reflect real-world conditions with information asymmetry. Our findings indicate that addressing information asymmetry remains a fundamental challenge for LLM-based agents.