The BURCHAK corpus: a Challenge Data Set for Interactive Learning of Visually Grounded Word Meanings

TL;DR

The paper introduces the BURCHAK corpus, a new dialogue dataset for studying interactive learning of visual words, along with a simulation framework and a reinforcement learning agent for visual word acquisition.

Contribution

It provides a novel dialogue dataset, a generic n-gram user simulation framework, and demonstrates RL-based learning of visual words from naturalistic interactions.

Findings

User simulations achieve 78% turn match similarity.

RL agent performs comparably to rule-based systems.

Dataset captures natural dialogue phenomena.

Abstract

We motivate and describe a new freely available human-human dialogue dataset for interactive learning of visually grounded word meanings through ostensive definition by a tutor to a learner. The data has been collected using a novel, character-by-character variant of the DiET chat tool (Healey et al., 2003; Mills and Healey, submitted) with a novel task, where a Learner needs to learn invented visual attribute words (such as " burchak " for square) from a tutor. As such, the text-based interactions closely resemble face-to-face conversation and thus contain many of the linguistic phenomena encountered in natural, spontaneous dialogue. These include self-and other-correction, mid-sentence continuations, interruptions, overlaps, fillers, and hedges. We also present a generic n-gram framework for building user (i.e. tutor) simulations from this type of incremental data, which is freely available to researchers. We show that the simulations produce outputs that are similar to the original data (e.g. 78% turn match similarity). Finally, we train and evaluate a Reinforcement Learning dialogue control agent for learning visually grounded word meanings, trained from the BURCHAK corpus. The learned policy shows comparable performance to a rule-based system built previously.