Emotion and Intent Joint Understanding in Multimodal Conversation: A Benchmarking Dataset

TL;DR

This paper introduces MC-EIU, a comprehensive multimodal dataset with emotion and intent annotations in English and Mandarin, and proposes the EI$^2$ network for joint understanding, demonstrating its effectiveness through experiments.

Contribution

It provides the first rich, open-source multimodal dataset for emotion and intent joint understanding in conversations and develops a novel deep correlation model for this task.

Findings

EI$^2$ outperforms baseline models in experiments

The dataset covers multiple modalities, languages, and categories

Ablation studies confirm the importance of multimodal features

Abstract

Emotion and Intent Joint Understanding in Multimodal Conversation (MC-EIU) aims to decode the semantic information manifested in a multimodal conversational history, while inferring the emotions and intents simultaneously for the current utterance. MC-EIU is enabling technology for many human-computer interfaces. However, there is a lack of available datasets in terms of annotation, modality, language diversity, and accessibility. In this work, we propose an MC-EIU dataset, which features 7 emotion categories, 9 intent categories, 3 modalities, i.e., textual, acoustic, and visual content, and two languages, i.e., English and Mandarin. Furthermore, it is completely open-source for free access. To our knowledge, MC-EIU is the first comprehensive and rich emotion and intent joint understanding dataset for multimodal conversation. Together with the release of the dataset, we also develop an Emotion and Intent Interaction (EI$^2$) network as a reference system by modeling the deep correlation between emotion and intent in the multimodal conversation. With comparative experiments and ablation studies, we demonstrate the effectiveness of the proposed EI$^2$ method on the MC-EIU dataset. The dataset and codes will be made available at: https://github.com/MC-EIU/MC-EIU.