Anti-efficient encoding in emergent communication

TL;DR

This paper investigates emergent communication in neural networks, revealing an anti-efficient encoding scheme that contrasts with human language patterns, and shows how different pressures influence the development of communication codes.

Contribution

It demonstrates that neural networks can develop anti-efficient codes and highlights the importance of functional pressures in shaping language-like properties.

Findings

Networks develop anti-efficient encoding schemes.

Including a message length penalty induces Zipf's Law of Abbreviation.

Anti-efficient codes may be easier for listeners to discriminate.

Abstract

Despite renewed interest in emergent language simulations with neural networks, little is known about the basic properties of the induced code, and how they compare to human language. One fundamental characteristic of the latter, known as Zipf's Law of Abbreviation (ZLA), is that more frequent words are efficiently associated to shorter strings. We study whether the same pattern emerges when two neural networks, a "speaker" and a "listener", are trained to play a signaling game. Surprisingly, we find that networks develop an \emph{anti-efficient} encoding scheme, in which the most frequent inputs are associated to the longest messages, and messages in general are skewed towards the maximum length threshold. This anti-efficient code appears easier to discriminate for the listener, and, unlike in human communication, the speaker does not impose a contrasting least-effort pressure towards brevity. Indeed, when the cost function includes a penalty for longer messages, the resulting message distribution starts respecting ZLA. Our analysis stresses the importance of studying the basic features of emergent communication in a highly controlled setup, to ensure the latter will not strand too far from human language. Moreover, we present a concrete illustration of how different functional pressures can lead to successful communication codes that lack basic properties of human language, thus highlighting the role such pressures play in the latter.