Emergent Communication between Heterogeneous Visual Agents through Decentralized Learning

TL;DR

This paper investigates how heterogeneous visual agents can develop shared symbols through decentralized learning without shared perception, using a novel game setup and experiments on MS-COCO.

Contribution

It introduces the Metropolis-Hastings Captioning Game to study emergent communication between agents with different visual representations, highlighting the role of perceptual similarity.

Findings

Shared token sequences are visually informative and outperform no-communication baselines.

Cross-agent alignment decreases with increasing encoder mismatch.

Encoder heterogeneity affects the number, specificity, and symmetry of emergent symbols.

Abstract

Symbols are shared, but perception is private. We study emergent communication between heterogeneous visual agents through decentralized learning, asking what visual information can become shareable when agents have different visual representations. Instead of optimizing messages through a shared external communicative objective, our agents exchange only discrete token sequences and update their own models using local perceptual evidence. This setting focuses on an underexplored aspect of emergent communication, examining whether common symbols can arise without shared perceptual access, and how the similarity between private visual spaces constrains the content and symmetry of the resulting language. We instantiate this setting in the Metropolis-Hastings Captioning Game (MHCG), where two agents collaboratively form shared captions by exchanging proposed token sequences that a listener accepts or rejects using an MH-style criterion evaluated against its own visual features. We compare three pairings of frozen visual encoders, with agents starting from randomly initialized text modules. Experiments on MS-COCO show that MHCG produces visually informative shared token sequences that outperform a no-communication baseline in cross-agent alignment, visual-feature prediction, and image-text retrieval; all cross-agent metrics decline as encoder mismatch increases. Moderate encoder heterogeneity reduces the number of shared sequences while preserving per-sequence visual specificity, whereas stronger encoder heterogeneity yields fewer, coarser, and more asymmetric sequences. Ablations show that listener-side MH acceptance is critical for avoiding degenerate token formation. These results suggest that shared symbols can arise from local perceptual evaluation alone, with visual representational similarity across encoders shaping both the content and symmetry of the resulting language.