PersuasiveToM: A Benchmark for Evaluating Machine Theory of Mind in Persuasive Dialogues

TL;DR

PersuasiveToM is a new benchmark designed to evaluate the Theory of Mind capabilities of large language models in complex persuasive dialogues, focusing on mental state tracking and application in social interactions.

Contribution

It introduces a comprehensive benchmark with two core tasks to assess LLMs' ToM reasoning and application in realistic persuasive conversations, addressing limitations of previous simplified tests.

Findings

Models perform well on basic questions but struggle with dynamic mental state tracking.

The benchmark reveals gaps in LLMs' understanding of complex psychological activities.

It provides a new standard for evaluating ToM in social and persuasive contexts.

Abstract

The ability to understand and predict the mental states of oneself and others, known as the Theory of Mind (ToM), is crucial for effective social scenarios. Although recent studies have evaluated ToM in Large Language Models (LLMs), existing benchmarks focus on simplified settings (e.g., Sally-Anne-style tasks) and overlook the complexity of real-world social interactions. To mitigate this gap, we propose PersuasiveToM, a benchmark designed to evaluate the ToM abilities of LLMs in persuasive dialogues. Our framework contains two core tasks: ToM Reasoning, which tests tracking of evolving desires, beliefs, and intentions; and ToM Application, which assesses the use of inferred mental states to predict and evaluate persuasion strategies. Experiments across eight leading LLMs reveal that while models excel on multiple questions, they struggle with the tasks that need tracking the dynamics and shifts of mental states and understanding the mental states in the whole dialogue comprehensively. Our aim with PersuasiveToM is to allow an effective evaluation of the ToM reasoning ability of LLMs with more focus on complex psychological activities. Our code is available at https://github.com/Yu-Fangxu/PersuasiveToM.