The Curious Case of Representational Alignment: Unravelling Visio-Linguistic Tasks in Emergent Communication

TL;DR

This paper investigates how representational alignment affects emergent communication in visio-linguistic tasks, revealing that increased alignment does not necessarily lead to human-like conceptual encoding or improved compositionality.

Contribution

It introduces an alignment penalty to prevent representational drift and analyzes its effects on emergent language properties and alignment metrics.

Findings

Emergent language does not encode human-like visual features.

Inter-agent alignment correlates with topographic similarity.

Alignment penalty prevents drift but does not enhance compositionality.

Abstract

Natural language has the universal properties of being compositional and grounded in reality. The emergence of linguistic properties is often investigated through simulations of emergent communication in referential games. However, these experiments have yielded mixed results compared to similar experiments addressing linguistic properties of human language. Here we address representational alignment as a potential contributing factor to these results. Specifically, we assess the representational alignment between agent image representations and between agent representations and input images. Doing so, we confirm that the emergent language does not appear to encode human-like conceptual visual features, since agent image representations drift away from inputs whilst inter-agent alignment increases. We moreover identify a strong relationship between inter-agent alignment and topographic similarity, a common metric for compositionality, and address its consequences. To address these issues, we introduce an alignment penalty that prevents representational drift but interestingly does not improve performance on a compositional discrimination task. Together, our findings emphasise the key role representational alignment plays in simulations of language emergence.