Can Machines Think Like Humans? A Behavioral Evaluation of LLM Agents in Dictator Games

TL;DR

This paper evaluates how Large Language Model agents behave prosocially in dictator games, comparing their actions to humans and analyzing the influence of personas and prompts on their decision-making.

Contribution

It introduces a social science framework to assess LLM agents' prosocial behaviors and systematically benchmarks their decision-making against human behaviors across different models.

Findings

Assigning human-like identities does not induce human-like prosocial behaviors in LLMs.

LLM behaviors vary significantly across architectures and prompts, with no clear pattern.

LLMs' reasoning in dictator games lacks consistent textual markers of human decision-making.

Abstract

As Large Language Model (LLM)-based agents increasingly engage with human society, how well do we understand their prosocial behaviors? We (1) investigate how LLM agents' prosocial behaviors can be induced by different personas and benchmarked against human behaviors; and (2) introduce a social science approach to evaluate LLM agents' decision-making. We explored how different personas and experimental framings affect these AI agents' altruistic behavior in dictator games and compared their behaviors within the same LLM family, across various families, and with human behaviors. The findings reveal that merely assigning a human-like identity to LLMs does not produce human-like behaviors. These findings suggest that LLM agents' reasoning does not consistently exhibit textual markers of human decision-making in dictator games and that their alignment with human behavior varies substantially across model architectures and prompt formulations; even worse, such dependence does not follow a clear pattern. As society increasingly integrates machine intelligence, "Prosocial AI" emerges as a promising and urgent research direction in philanthropic studies.