BeliefNest: A Joint Action Simulator for Embodied Agents with Theory of Mind

TL;DR

BeliefNest is an open-source Minecraft-based simulator that enables embodied agents to perform collaborative tasks by explicitly modeling nested belief states and reasoning about others' beliefs using large language models.

Contribution

It introduces BeliefNest, a hierarchical simulator that allows agents to explicitly represent and reason about nested beliefs in open-domain tasks.

Findings

Agents can infer others' beliefs accurately.

Agents can predict belief-based actions in false-belief tasks.

BeliefNest facilitates the development of Theory of Mind in embodied agents.

Abstract

This paper introduces an open-source simulator, BeliefNest, designed to enable embodied agents to perform collaborative tasks by leveraging Theory of Mind. BeliefNest dynamically and hierarchically constructs simulators within a Minecraft environment, allowing agents to explicitly represent nested belief states about themselves and others. This enables agent control in open-domain tasks that require Theory of Mind reasoning. The simulator provides a prompt generation mechanism based on each belief state, facilitating the design and evaluation of methods for agent control utilizing large language models (LLMs). We demonstrate through experiments that agents can infer others' beliefs and predict their belief-based actions in false-belief tasks.