Learning to Communicate Across Modalities: Perceptual Heterogeneity in Multi-Agent Systems

TL;DR

This paper investigates how multi-agent systems with different perceptual modalities develop communication protocols, revealing differences in efficiency, encoding, and transferability of learned representations across heterogeneous perceptual worlds.

Contribution

It introduces a multi-step binary communication game with perceptual heterogeneity, demonstrating how agents adapt and transfer emergent communication across diverse modalities.

Findings

Multimodal systems converge to class-consistent, perceptually grounded messages.

Unimodal systems communicate more efficiently with fewer bits.

Limited fine-tuning enables cross-system communication despite perceptual differences.

Abstract

Emergent communication offers insight into how agents develop shared structured representations, yet most research assumes homogeneous modalities or aligned representational spaces, overlooking the perceptual heterogeneity of real-world settings. We study a heterogeneous multi-step binary communication game where agents differ in modality and lack perceptual grounding. Despite perceptual misalignment, multimodal systems converge to class-consistent messages grounded in perceptual input. Unimodal systems communicate more efficiently, using fewer bits and achieving lower classification entropy, while multimodal agents require greater information exchange and exhibit higher uncertainty. Bit perturbation experiments provide strong evidence that meaning is encoded in a distributional rather than compositional manner, as each bit's contribution depends on its surrounding pattern. Finally, interoperability analyses show that systems trained in different perceptual worlds fail to directly communicate, but limited fine-tuning enables successful cross-system communication. This work positions emergent communication as a framework for studying how agents adapt and transfer representations across heterogeneous modalities, opening new directions for both theory and experimentation.