The emergence of visual semantics through communication games
Daniela Mihai, Jonathon Hare

TL;DR
This paper investigates how agents can develop semantically meaningful visual communication without pretraining, by exploring the effects of inductive biases and self-supervised learning in referential signaling games with realistic images.
Contribution
It demonstrates that visual semantics can emerge in communication systems through self-supervised learning and input augmentations, reducing reliance on pretrained feature extractors.
Findings
Pretrained networks bias the visual semantics learned by agents.
Input augmentations and additional tasks induce more meaningful visual representations.
Semantic communication can be achieved without supervised pretraining.
Abstract
The emergence of communication systems between agents which learn to play referential signalling games with realistic images has attracted a lot of attention recently. The majority of work has focused on using fixed, pretrained image feature extraction networks which potentially bias the information the agents learn to communicate. In this work, we consider a signalling game setting in which a `sender' agent must communicate the information about an image to a `receiver' who must select the correct image from many distractors. We investigate the effect of the feature extractor's weights and of the task being solved on the visual semantics learned by the models. We first demonstrate to what extent the use of pretrained feature extraction networks inductively bias the visual semantics conveyed by emergent communication channel and quantify the visual semantics that are induced. We then…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsLanguage and cultural evolution · Multimodal Machine Learning Applications · Reinforcement Learning in Robotics
