Position: Theory of Mind Benchmarks are Broken for Large Language Models

TL;DR

This paper critiques current theory of mind benchmarks for large language models, highlighting their inability to test models' adaptation to new partners and proposing a new concept called functional theory of mind.

Contribution

The paper introduces the concept of functional theory of mind, emphasizing the importance of evaluating LLMs' ability to adapt rationally in context, beyond predicting behavior.

Findings

Many open source LLMs show strong literal theory of mind.

LLMs struggle with functional theory of mind, especially over long interactions.

Literal theory of mind performance does not imply functional theory of mind capabilities.

Abstract

Our paper argues that the majority of theory of mind benchmarks are broken because of their inability to directly test how large language models (LLMs) adapt to new partners. This problem stems from the fact that theory of mind benchmarks for LLMs are overwhelmingly inspired by the methods used to test theory of mind in humans and fall victim to a fallacy of attributing human-like qualities to AI agents. We expect that humans will engage in a consistent reasoning process across various questions about a situation, but this is known to not be the case for current LLMs. Most theory of mind benchmarks only measure what we call literal theory of mind: the ability to predict the behavior of others. However, this type of metric is only informative when agents exhibit self-consistent reasoning. Thus, we introduce the concept of functional theory of mind: the ability to adapt to agents in-context following a rational response to their behavior. We find that many open source LLMs are capable of displaying strong literal theory of mind capabilities, but seem to struggle with functional theory of mind -- even with exceedingly simple partner policies. Simply put, strong literal theory of mind performance does not necessarily imply strong functional theory of mind performance or vice versa. Achieving functional theory of mind, particularly over long interaction horizons with a partner, is a significant challenge deserving a prominent role in any meaningful LLM theory of mind evaluation.