Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols

TL;DR

This paper investigates how two agents can develop a natural language-like communication protocol from scratch through a referential game, comparing reinforcement learning and differentiable methods, and exploring properties like compositionality and variability.

Contribution

It introduces a multi-agent game framework where agents learn to communicate with sequences of symbols, demonstrating faster convergence and more natural language properties with differentiable training methods.

Findings

Differentiable relaxation methods converge faster than reinforcement learning.

The emergent communication protocol shows compositionality and variability.

Injecting prior language knowledge influences protocol properties.

Abstract

Learning to communicate through interaction, rather than relying on explicit supervision, is often considered a prerequisite for developing a general AI. We study a setting where two agents engage in playing a referential game and, from scratch, develop a communication protocol necessary to succeed in this game. Unlike previous work, we require that messages they exchange, both at train and test time, are in the form of a language (i.e. sequences of discrete symbols). We compare a reinforcement learning approach and one using a differentiable relaxation (straight-through Gumbel-softmax estimator) and observe that the latter is much faster to converge and it results in more effective protocols. Interestingly, we also observe that the protocol we induce by optimizing the communication success exhibits a degree of compositionality and variability (i.e. the same information can be phrased in different ways), both properties characteristic of natural languages. As the ultimate goal is to ensure that communication is accomplished in natural language, we also perform experiments where we inject prior information about natural language into our model and study properties of the resulting protocol.