PatchGame: Learning to Signal Mid-level Patches in Referential Games

TL;DR

PatchGame introduces a referential game where agents learn to communicate important image patches, enabling faster vision transformer training and effective pre-training for recognition tasks without supervision.

Contribution

The paper presents a novel signaling game framework for agents to develop communication protocols based on image patches without supervision.

Findings

Agents successfully develop communication protocols for image patches.

Using important patches accelerates Vision Transformer training.

Pre-training with PatchGame improves downstream recognition performance.

Abstract

We study a referential game (a type of signaling game) where two agents communicate with each other via a discrete bottleneck to achieve a common goal. In our referential game, the goal of the speaker is to compose a message or a symbolic representation of "important" image patches, while the task for the listener is to match the speaker's message to a different view of the same image. We show that it is indeed possible for the two agents to develop a communication protocol without explicit or implicit supervision. We further investigate the developed protocol and show the applications in speeding up recent Vision Transformers by using only important patches, and as pre-training for downstream recognition tasks (e.g., classification). Code available at https://github.com/kampta/PatchGame.