Effects of Theory of Mind and Prosocial Beliefs on Steering Human-Aligned Behaviors of LLMs in Ultimatum Games

TL;DR

This paper explores how Theory of Mind reasoning and prosocial beliefs in large language models influence their ability to align with human norms in negotiation tasks like the ultimatum game, showing that ToM enhances decision-making and cooperation.

Contribution

It demonstrates the impact of ToM reasoning and prosocial beliefs on LLM behavior in social negotiations, providing empirical evidence across multiple models and simulation scenarios.

Findings

ToM reasoning improves behavior alignment and decision consistency.

Models with ToM reasoning outperform reasoning-only models.

Different ToM levels influence negotiation outcomes.

Abstract

Large Language Models (LLMs) have shown potential in simulating human behaviors and performing theory-of-mind (ToM) reasoning, a crucial skill for complex social interactions. In this study, we investigate the role of ToM reasoning in aligning agentic behaviors with human norms in negotiation tasks, using the ultimatum game as a controlled environment. We initialized LLM agents with different prosocial beliefs (including Greedy, Fair, and Selfless) and reasoning methods like chain-of-thought (CoT) and varying ToM levels, and examined their decision-making processes across diverse LLMs, including reasoning models like o3-mini and DeepSeek-R1 Distilled Qwen 32B. Results from 2,700 simulations indicated that ToM reasoning enhances behavior alignment, decision-making consistency, and negotiation outcomes. Consistent with previous findings, reasoning models exhibit limited capability compared to models with ToM reasoning, different roles of the game benefits with different orders of ToM reasoning. Our findings contribute to the understanding of ToM's role in enhancing human-AI interaction and cooperative decision-making. The code used for our experiments can be found at https://github.com/Stealth-py/UltimatumToM.