Do Large Language Models Possess a Theory of Mind? A Comparative Evaluation Using the Strange Stories Paradigm

TL;DR

This study evaluates whether large language models can infer mental states like beliefs and emotions, comparing their performance to humans using a standard ToM test, revealing GPT-4o's near-human accuracy.

Contribution

It provides a comparative analysis of LLMs' Theory of Mind capabilities, highlighting GPT-4o's robustness and performance close to human levels in mental-state inference tasks.

Findings

GPT-4o performs comparably to humans in ToM tasks.

Smaller models are sensitive to inferential cues and distractions.

Performance varies significantly across different LLMs.

Abstract

The study explores whether current Large Language Models (LLMs) exhibit Theory of Mind (ToM) capabilities -- specifically, the ability to infer others' beliefs, intentions, and emotions from text. Given that LLMs are trained on language data without social embodiment or access to other manifestations of mental representations, their apparent social-cognitive reasoning raises key questions about the nature of their understanding. Are they capable of robust mental-state attribution indistinguishable from human ability in its output, or do their outputs merely reflect superficial pattern completion? To address this question, we tested five LLMs and compared their performance to that of human controls using an adapted version of a text-based tool widely used in human ToM research. The test involves answering questions about the beliefs, intentions, and emotions of story characters. The results revealed a performance gap between the models. Earlier and smaller models were strongly affected by the number of relevant inferential cues available and, to some extent, were also vulnerable to the presence of irrelevant or distracting information in the texts. In contrast, GPT-4o demonstrated high accuracy and strong robustness, performing comparably to humans even in the most challenging conditions. This work contributes to ongoing debates about the cognitive status of LLMs and the boundary between genuine understanding and statistical approximation.