FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions
Hyunwoo Kim, Melanie Sclar, Xuhui Zhou, Ronan Le Bras, Gunhee Kim,, Yejin Choi, Maarten Sap

TL;DR
FANToM is a new benchmark designed to evaluate the reasoning capabilities of large language models in interactive, information-asymmetric scenarios, revealing their limitations in understanding theory of mind.
Contribution
This paper introduces FANToM, a novel benchmark that tests LLMs' theory of mind in interactive contexts, incorporating psychological principles and diverse question types.
Findings
LLMs perform poorly on FANToM compared to humans.
Chain-of-thought reasoning and fine-tuning do not fully bridge the performance gap.
FANToM exposes limitations of current LLMs in understanding complex social reasoning.
Abstract
Theory of mind (ToM) evaluations currently focus on testing models using passive narratives that inherently lack interactivity. We introduce FANToM, a new benchmark designed to stress-test ToM within information-asymmetric conversational contexts via question answering. Our benchmark draws upon important theoretical requisites from psychology and necessary empirical considerations when evaluating large language models (LLMs). In particular, we formulate multiple types of questions that demand the same underlying reasoning to identify illusory or false sense of ToM capabilities in LLMs. We show that FANToM is challenging for state-of-the-art LLMs, which perform significantly worse than humans even with chain-of-thought reasoning or fine-tuning.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Social Robot Interaction and HRI · Speech and dialogue systems
MethodsFocus
