Effects of Theory of Mind and Prosocial Beliefs on Steering Human-Aligned Behaviors of LLMs in Ultimatum Games
Neemesh Yadav, Palakorn Achananuparp, Jing Jiang, Ee-Peng Lim

TL;DR
This paper explores how Theory of Mind reasoning and prosocial beliefs in large language models influence their ability to align with human norms in negotiation tasks like the ultimatum game, showing that ToM enhances decision-making and cooperation.
Contribution
It demonstrates the impact of ToM reasoning and prosocial beliefs on LLM behavior in social negotiations, providing empirical evidence across multiple models and simulation scenarios.
Findings
ToM reasoning improves behavior alignment and decision consistency.
Models with ToM reasoning outperform reasoning-only models.
Different ToM levels influence negotiation outcomes.
Abstract
Large Language Models (LLMs) have shown potential in simulating human behaviors and performing theory-of-mind (ToM) reasoning, a crucial skill for complex social interactions. In this study, we investigate the role of ToM reasoning in aligning agentic behaviors with human norms in negotiation tasks, using the ultimatum game as a controlled environment. We initialized LLM agents with different prosocial beliefs (including Greedy, Fair, and Selfless) and reasoning methods like chain-of-thought (CoT) and varying ToM levels, and examined their decision-making processes across diverse LLMs, including reasoning models like o3-mini and DeepSeek-R1 Distilled Qwen 32B. Results from 2,700 simulations indicated that ToM reasoning enhances behavior alignment, decision-making consistency, and negotiation outcomes. Consistent with previous findings, reasoning models exhibit limited capability…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsEvolutionary Game Theory and Cooperation · Opinion Dynamics and Social Influence
