Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models

TL;DR

This paper introduces VisionToM, a framework that enhances multimodal large language models' ability to infer human mental states from visual data, improving reasoning, interpretability, and alignment in real-world scenarios.

Contribution

We propose VisionToM, a novel intervention method that aligns visual representations with semantic targets, boosting ToM reasoning and interpretability in multimodal models.

Findings

Significant improvement in ToM accuracy on EgoToM benchmark

Enhanced interpretability through attention analysis

Better free-form explanation quality in open-ended tasks

Abstract

As large language models (LLMs) continue to advance, there is increasing interest in their ability to infer human mental states and demonstrate a human-like Theory of Mind (ToM). Most existing ToM evaluations, however, are centered on text-based inputs, while scenarios relying solely on visual information receive far less attention. This leaves a gap, since real-world human-AI interaction typically requires multimodal understanding. In addition, many current methods regard the model as a black box and rarely probe how its internal attention behaves in multiple-choice question answering (QA). The impact of LLM hallucinations on such tasks is also underexplored from an interpretability perspective. To address these issues, we introduce VisionToM, a vision-oriented intervention framework designed to strengthen task-aware reasoning. The core idea is to compute intervention vectors that align visual representations with the correct semantic targets, thereby steering the model's attention through different layers of visual features. This guidance reduces the model's reliance on spurious linguistic priors, leading to more reliable multimodal language model (MLLM) outputs and better QA performance. Experiments on the EgoToM benchmark-an egocentric, real-world video dataset for ToM with three multiple-choice QA settings-demonstrate that our method substantially improves the ToM abilities of MLLMs. Furthermore, results on an additional open-ended generation task show that VisionToM enables MLLMs to produce free-form explanations that more accurately capture agents' mental states, pushing machine-human collaboration toward greater alignment.