Investigating the Development of Task-Oriented Communication in Vision-Language Models

TL;DR

This paper explores how vision-language models can develop task-specific communication protocols that are efficient and covert, highlighting both their potential and associated risks in collaborative reasoning tasks.

Contribution

It demonstrates that VLM agents can create effective, task-adapted, and covert communication protocols within a referential-game framework, revealing new insights into AI communication behaviors.

Findings

VLMs develop efficient, task-specific communication patterns.

VLMs can create covert protocols difficult for humans to interpret.

Spontaneous coordination occurs without explicit shared protocols.

Abstract

We investigate whether \emph{LLM-based agents} can develop task-oriented communication protocols that differ from standard natural language in collaborative reasoning tasks. Our focus is on two core properties such task-oriented protocols may exhibit: Efficiency -- conveying task-relevant information more concisely than natural language, and Covertness -- becoming difficult for external observers to interpret, raising concerns about transparency and control. To investigate these aspects, we use a referential-game framework in which vision-language model (VLM) agents communicate, providing a controlled, measurable setting for evaluating language variants. Experiments show that VLMs can develop effective, task-adapted communication patterns. At the same time, they can develop covert protocols that are difficult for humans and external agents to interpret. We also observe spontaneous coordination between similar models without explicitly shared protocols. These findings highlight both the potential and the risks of task-oriented communication, and position referential games as a valuable testbed for future work in this area.