Shared Visual Representations of Drawing for Communication: How do different biases affect human interpretability and intent?

TL;DR

This paper explores how biases in visual representations influence the interpretability and intent of drawings produced by artificial agents in communication tasks, highlighting the role of pretrained encoders and semantic analysis.

Contribution

It demonstrates that combining pretrained encoders with inductive biases enables agents to produce recognizable sketches that communicate effectively, and introduces methods to analyze the semantic content of sketches.

Findings

Powerful pretrained encoders improve sketch recognizability.

Inductive biases influence the emergence of objectness in sketches.

Semantic analysis reveals objectness as a key feature in self-supervised training.

Abstract

We present an investigation into how representational losses can affect the drawings produced by artificial agents playing a communication game. Building upon recent advances, we show that a combination of powerful pretrained encoder networks, with appropriate inductive biases, can lead to agents that draw recognisable sketches, whilst still communicating well. Further, we start to develop an approach to help automatically analyse the semantic content being conveyed by a sketch and demonstrate that current approaches to inducing perceptual biases lead to a notion of objectness being a key feature despite the agent training being self-supervised.