Spot the Difference: A Cooperative Object-Referring Game in Non-Perfectly Co-Observable Scene

TL;DR

This paper introduces a new dialog task in non-perfectly co-observable visual scenes, along with a large-scale dataset and benchmark models, to advance research in realistic visual dialog scenarios involving scene differences.

Contribution

It proposes a novel object-referring game in non-co-observable scenes, creates the SpotDiff dataset with VR images and dialogs, and provides benchmark models and analysis.

Findings

Benchmark models achieve baseline performance.

Identified key challenges in dialog strategy and object categorization.

Dataset enables future research in realistic visual dialog scenarios.

Abstract

Visual dialog has witnessed great progress after introducing various vision-oriented goals into the conversation, especially such as GuessWhich and GuessWhat, where the only image is visible by either and both of the questioner and the answerer, respectively. Researchers explore more on visual dialog tasks in such kind of single- or perfectly co-observable visual scene, while somewhat neglect the exploration on tasks of non perfectly co-observable visual scene, where the images accessed by two agents may not be exactly the same, often occurred in practice. Although building common ground in non-perfectly co-observable visual scene through conversation is significant for advanced dialog agents, the lack of such dialog task and corresponding large-scale dataset makes it impossible to carry out in-depth research. To break this limitation, we propose an object-referring game in non-perfectly co-observable visual scene, where the goal is to spot the difference between the similar visual scenes through conversing in natural language. The task addresses challenges of the dialog strategy in non-perfectly co-observable visual scene and the ability of categorizing objects. Correspondingly, we construct a large-scale multimodal dataset, named SpotDiff, which contains 87k Virtual Reality images and 97k dialogs generated by self-play. Finally, we give benchmark models for this task, and conduct extensive experiments to evaluate its performance as well as analyze its main challenges.