Interpretation of Emergent Communication in Heterogeneous Collaborative Embodied Agents

TL;DR

This paper introduces the CoMON task to analyze how emergent communication between heterogeneous embodied AI agents can be grounded in perception and spatial structure, revealing interpretability in their communication patterns.

Contribution

It proposes a new collaborative navigation task and analyzes emergent communication, demonstrating grounding in perception and spatial environment structure.

Findings

Emergent communication is grounded in agent observations.

Communication patterns reflect spatial and egocentric perspectives.

Heterogeneous agents develop interpretable communication strategies.

Abstract

Communication between embodied AI agents has received increasing attention in recent years. Despite its use, it is still unclear whether the learned communication is interpretable and grounded in perception. To study the grounding of emergent forms of communication, we first introduce the collaborative multi-object navigation task CoMON. In this task, an oracle agent has detailed environment information in the form of a map. It communicates with a navigator agent that perceives the environment visually and is tasked to find a sequence of goals. To succeed at the task, effective communication is essential. CoMON hence serves as a basis to study different communication mechanisms between heterogeneous agents, that is, agents with different capabilities and roles. We study two common communication mechanisms and analyze their communication patterns through an egocentric and spatial lens. We show that the emergent communication can be grounded to the agent observations and the spatial structure of the 3D environment. Video summary: https://youtu.be/kLv2rxO9t0g