An implementation of the "Guess who?" game using CLIP

TL;DR

This paper demonstrates how CLIP can be used to implement the game 'Guess who?' by allowing natural language interaction and automatic image recognition, while analyzing its zero-shot performance and limitations.

Contribution

The work applies CLIP to a game setting, showcasing its capabilities and limitations in a practical, interactive application.

Findings

CLIP effectively interprets natural language prompts in the game.

Zero-shot performance varies with different prompting strategies.

Limitations of CLIP's zero-shot abilities are identified.

Abstract

CLIP (Contrastive Language-Image Pretraining) is an efficient method for learning computer vision tasks from natural language supervision that has powered a recent breakthrough in deep learning due to its zero-shot transfer capabilities. By training from image-text pairs available on the internet, the CLIP model transfers non-trivially to most tasks without the need for any data set specific training. In this work, we use CLIP to implement the engine of the popular game "Guess who?", so that the player interacts with the game using natural language prompts and CLIP automatically decides whether an image in the game board fulfills that prompt or not. We study the performance of this approach by benchmarking on different ways of prompting the questions to CLIP, and show the limitations of its zero-shot capabilites.