Emergent Communication in a Multi-Modal, Multi-Step Referential Game

TL;DR

This paper introduces a multi-modal, multi-step referential game where agents develop a natural language-like communication protocol to identify objects, demonstrating improved accuracy and generalization with increased communication bandwidth.

Contribution

It presents a novel multi-modal, multi-step referential game framework that enables agents to develop more natural and flexible communication protocols.

Findings

Emergent communication closely resembles natural language.

Increased communication bandwidth enhances generalization.

Gradual information exchange improves prediction accuracy.

Abstract

Inspired by previous work on emergent communication in referential games, we propose a novel multi-modal, multi-step referential game, where the sender and receiver have access to distinct modalities of an object, and their information exchange is bidirectional and of arbitrary duration. The multi-modal multi-step setting allows agents to develop an internal communication significantly closer to natural language, in that they share a single set of messages, and that the length of the conversation may vary according to the difficulty of the task. We examine these properties empirically using a dataset consisting of images and textual descriptions of mammals, where the agents are tasked with identifying the correct object. Our experiments indicate that a robust and efficient communication protocol emerges, where gradual information exchange informs better predictions and higher communication bandwidth improves generalization.