GOMA: Proactive Embodied Cooperative Communication via Goal-Oriented Mental Alignment

TL;DR

GOMA introduces a goal-oriented mental alignment framework enabling embodied assistants to proactively communicate via natural language, improving cooperation in complex environments by reducing mental state misalignment.

Contribution

This paper presents a novel planning-based approach for proactive verbal communication that aligns agents' mental states to enhance cooperation.

Findings

GOMA outperforms strong baselines in cooperative tasks.

Large language models struggle with contextually meaningful communication.

Proactive communication improves human perception and task performance.

Abstract

Verbal communication plays a crucial role in human cooperation, particularly when the partners only have incomplete information about the task, environment, and each other's mental state. In this paper, we propose a novel cooperative communication framework, Goal-Oriented Mental Alignment (GOMA). GOMA formulates verbal communication as a planning problem that minimizes the misalignment between the parts of agents' mental states that are relevant to the goals. This approach enables an embodied assistant to reason about when and how to proactively initialize communication with humans verbally using natural language to help achieve better cooperation. We evaluate our approach against strong baselines in two challenging environments, Overcooked (a multiplayer game) and VirtualHome (a household simulator). Our experimental results demonstrate that large language models struggle with generating meaningful communication that is grounded in the social and physical context. In contrast, our approach can successfully generate concise verbal communication for the embodied assistant to effectively boost the performance of the cooperation as well as human users' perception of the assistant.