Affective Visual Dialog: A Large-Scale Benchmark for Emotional Reasoning Based on Visually Grounded Conversations

TL;DR

This paper presents Affective Visual Dialog, a large-scale dataset and benchmark for emotional reasoning in visually grounded conversations, enabling research on emotion understanding, prediction, and explanation.

Contribution

It introduces the AffectVisDial dataset with 50K dialogs and develops baseline models for emotional reasoning tasks in visual conversations.

Findings

Models show promising emotional reasoning abilities.

The dataset enables multi-skill emotion understanding.

Baseline models outperform random guessing.

Abstract

We introduce Affective Visual Dialog, an emotion explanation and reasoning task as a testbed for research on understanding the formation of emotions in visually grounded conversations. The task involves three skills: (1) Dialog-based Question Answering (2) Dialog-based Emotion Prediction and (3) Affective emotion explanation generation based on the dialog. Our key contribution is the collection of a large-scale dataset, dubbed AffectVisDial, consisting of 50K 10-turn visually grounded dialogs as well as concluding emotion attributions and dialog-informed textual emotion explanations, resulting in a total of 27,180 working hours. We explain our design decisions in collecting the dataset and introduce the questioner and answerer tasks that are associated with the participants in the conversation. We train and demonstrate solid Affective Visual Dialog baselines adapted from state-of-the-art models. Remarkably, the responses generated by our models show promising emotional reasoning abilities in response to visually grounded conversations. Our project page is available at https://affective-visual-dialog.github.io.