The Spot the Difference corpus: a multi-modal corpus of spontaneous task oriented spoken interactions

TL;DR

The paper introduces the Spot the Difference Corpus, a multi-modal dataset of spontaneous, task-oriented spoken interactions with rich annotations, enabling research on human communication, turn-taking, and referring expressions.

Contribution

It provides a comprehensive, multi-modal corpus of spontaneous dialogues with detailed annotations, facilitating diverse studies in human communication and interaction dynamics.

Findings

More complex scenes tend to have more interruptions.

Task success varies with scene complexity.

Scene entropy relates to speech overlaps and uncertainty.

Abstract

This paper describes the Spot the Difference Corpus which contains 54 interactions between pairs of subjects interacting to find differences in two very similar scenes. The setup used, the participants' metadata and details about collection are described. We are releasing this corpus of task-oriented spontaneous dialogues. This release includes rich transcriptions, annotations, audio and video. We believe that this dataset constitutes a valuable resource to study several dimensions of human communication that go from turn-taking to the study of referring expressions. In our preliminary analyses we have looked at task success (how many differences were found out of the total number of differences) and how it evolves over time. In addition we have looked at scene complexity provided by the RGB components' entropy and how it could relate to speech overlaps, interruptions and the expression of uncertainty. We found there is a tendency that more complex scenes have more competitive interruptions.