The Essence of Contextual Understanding in Theory of Mind: A Study on Question Answering with Story Characters

TL;DR

This study emphasizes the importance of comprehensive contextual understanding, especially personal backgrounds, for Theory of Mind tasks, revealing current LLMs' limitations compared to human performance in story-based reasoning.

Contribution

Introduces the CharToM benchmark with 1,035 questions based on classic novels to evaluate LLMs' ToM capabilities considering global context and personal backgrounds.

Findings

Humans outperform LLMs significantly when understanding character backgrounds.

LLMs perform worse than humans despite pre-training on story data.

Current LLMs struggle with nuanced contextual reasoning in ToM tasks.

Abstract

Theory-of-Mind (ToM) is a fundamental psychological capability that allows humans to understand and interpret the mental states of others. Humans infer others' thoughts by integrating causal cues and indirect clues from broad contextual information, often derived from past interactions. In other words, human ToM heavily relies on the understanding about the backgrounds and life stories of others. Unfortunately, this aspect is largely overlooked in existing benchmarks for evaluating machines' ToM capabilities, due to their usage of short narratives without global context, especially personal background of characters. In this paper, we verify the importance of comprehensive contextual understanding about personal backgrounds in ToM and assess the performance of LLMs in such complex scenarios. To achieve this, we introduce CharToM benchmark, comprising 1,035 ToM questions based on characters from classic novels. Our human study reveals a significant disparity in performance: the same group of educated participants performs dramatically better when they have read the novels compared to when they have not. In parallel, our experiments on state-of-the-art LLMs, including the very recent o1 and DeepSeek-R1 models, show that LLMs still perform notably worse than humans, despite that they have seen these stories during pre-training. This highlights the limitations of current LLMs in capturing the nuanced contextual information required for ToM reasoning.