OSCToM: RL-Guided Adversarial Generation for High-Order Theory of Mind

TL;DR

OSCToM introduces a reinforcement learning-based method for modeling nested belief conflicts in language models, significantly improving performance on complex Theory of Mind tasks with more efficient data synthesis.

Contribution

The paper presents OSCToM, a novel RL-guided approach combining domain-specific language and surrogate models to enhance recursive ToM reasoning in LLMs.

Findings

OSCToM-8B outperforms existing systems on ToM benchmarks.

Achieves 76% accuracy on FANToM, surpassing prior methods.

Data-synthesis is 6x more efficient, aiding smaller models.

Abstract

Large Language Models (LLMs) perform well on many language tasks, but their Theory of Mind (ToM) reasoning is still uneven in complex social settings. Existing benchmarks, including ExploreToM, do not always test the recursive beliefs and information asymmetries that make these settings difficult. This paper presents OSCToM (Observer-Self Conflict Theory of Mind), an approach for modeling nested belief conflicts in LLM-based ToM tasks. The key case is one in which an observer's view of another agent conflicts with the observer's own belief state. Such cases go beyond simple perspective-taking and require recursive, multi-layered reasoning. OSCToM combines reinforcement learning (RL), an extended domain-specific language, and compositional surrogate models to generate observer-self conflicts. In our experiments, OSCToM-8B gives the best overall result among the systems tested. It improves on the reported ExploreToM results on FANToM and remains competitive on Hi-ToM and BigToM. On the information-asymmetric FANToM benchmark, OSCToM reaches 76% accuracy, compared with the 0.2% reported by ExploreToM. The data-synthesis procedure is also 6x more efficient, indicating that targeted training data can help smaller models handle advanced cognitive reasoning. The project code is available at https://github.com/sharminsrishty/osct.