Game of Sketches: Deep Recurrent Models of Pictionary-style Word Guessing

TL;DR

This paper introduces a novel deep learning model for Pictionary-style word guessing using sketch data, combining elements of visual question answering and game simulation, with promising results in mimicking human guessing behavior.

Contribution

The work presents the first computational model for Pictionary, integrating sketch-based visual data with open-ended guessing, and introduces the Sketch-QA dataset for this task.

Findings

Model generates human-like guesses and mistakes.

Experimental results show promising accuracy and human mimicry.

The Sketch-QA dataset enables new research in sketch-based visual reasoning.

Abstract

The ability of intelligent agents to play games in human-like fashion is popularly considered a benchmark of progress in Artificial Intelligence. Similarly, performance on multi-disciplinary tasks such as Visual Question Answering (VQA) is considered a marker for gauging progress in Computer Vision. In our work, we bring games and VQA together. Specifically, we introduce the first computational model aimed at Pictionary, the popular word-guessing social game. We first introduce Sketch-QA, an elementary version of Visual Question Answering task. Styled after Pictionary, Sketch-QA uses incrementally accumulated sketch stroke sequences as visual data. Notably, Sketch-QA involves asking a fixed question ("What object is being drawn?") and gathering open-ended guess-words from human guessers. We analyze the resulting dataset and present many interesting findings therein. To mimic Pictionary-style guessing, we subsequently propose a deep neural model which generates guess-words in response to temporally evolving human-drawn sketches. Our model even makes human-like mistakes while guessing, thus amplifying the human mimicry factor. We evaluate our model on the large-scale guess-word dataset generated via Sketch-QA task and compare with various baselines. We also conduct a Visual Turing Test to obtain human impressions of the guess-words generated by humans and our model. Experimental results demonstrate the promise of our approach for Pictionary and similarly themed games.