Collecting Visually-Grounded Dialogue with A Game Of Sorts

TL;DR

This paper introduces 'A Game Of Sorts', a collaborative dialogue task for studying visually-grounded communication, emphasizing argumentation and cooperation over minimal expressions, with initial data collection and open resources.

Contribution

It presents a novel interactive game for collecting grounded dialogue data that captures the collaborative nature of referential communication.

Findings

Collected dialogue data involving argumentation and cooperation.

Provided open access to data, code, and application.

Demonstrated feasibility of the task through a small-scale experiment.

Abstract

An idealized, though simplistic, view of the referring expression production and grounding process in (situated) dialogue assumes that a speaker must merely appropriately specify their expression so that the target referent may be successfully identified by the addressee. However, referring in conversation is a collaborative process that cannot be aptly characterized as an exchange of minimally-specified referring expressions. Concerns have been raised regarding assumptions made by prior work on visually-grounded dialogue that reveal an oversimplified view of conversation and the referential process. We address these concerns by introducing a collaborative image ranking task, a grounded agreement game we call "A Game Of Sorts". In our game, players are tasked with reaching agreement on how to rank a set of images given some sorting criterion through a largely unrestricted, role-symmetric dialogue. By putting emphasis on the argumentation in this mixed-initiative interaction, we collect discussions that involve the collaborative referential process. We describe results of a small-scale data collection experiment with the proposed task. All discussed materials, which includes the collected data, the codebase, and a containerized version of the application, are publicly available.