MindPower: Enabling Theory-of-Mind Reasoning in VLM-based Embodied Agents

TL;DR

MindPower is a novel framework that enables vision-language embodied agents to perform theory-of-mind reasoning, improving decision-making and action generation by modeling both self and others' mental states.

Contribution

We introduce MindPower, a comprehensive framework integrating perception, mental reasoning, decision-making, and action for ToM in embodied agents, along with a new optimization objective.

Findings

Outperforms GPT-4o by 12.77% in decision making

Outperforms GPT-4o by 12.49% in action generation

Demonstrates effective modeling of self and others' mental states

Abstract

Theory of Mind (ToM) refers to the ability to infer others' mental states, such as beliefs, desires, and intentions. Current vision-language embodied agents lack ToM-based decision-making, and existing benchmarks focus solely on human mental states while ignoring the agent's own perspective, hindering coherent decision and action generation. To address this, we propose MindPower, a Robot-Centric framework integrating Perception, Mental Reasoning, Decision Making and Action. Given multimodal inputs, MindPower first perceives the environment and human states, then performs ToM Reasoning to model both self and others, and finally generates decisions and actions guided by inferred mental states. Furthermore, we introduce Mind-Reward, a novel optimization objective that encourages VLMs to produce consistent ToM Reasoning and behavior. Our model outperforms GPT-4o by 12.77% in decision making and 12.49% in action generation.