Generative Emotion Cause Explanation in Multimodal Conversations

TL;DR

This paper introduces a new multimodal task and dataset for explaining emotional causes in conversations, leveraging large language models and visual data to improve understanding of emotional triggers.

Contribution

It proposes the MECEC task, creates the ECEM dataset, and develops FAME-Net, a novel LLM-based model that analyzes visual cues for emotional cause explanation in multimodal conversations.

Findings

FAME-Net outperforms baseline models on the ECEM dataset.

The dataset combines video clips with detailed emotion explanations.

The approach effectively captures facial emotional contagion to identify emotional causes.

Abstract

Multimodal conversation, a crucial form of human communication, carries rich emotional content, making the exploration of the causes of emotions within it a research endeavor of significant importance. However, existing research on the causes of emotions typically employs an utterance selection method within a single textual modality to locate causal utterances. This approach remains limited to coarse-grained assessments, lacks nuanced explanations of emotional causation, and demonstrates inadequate capability in identifying multimodal emotional triggers. Therefore, we introduce a task-\textbf{Multimodal Emotion Cause Explanation in Conversation (MECEC)}. This task aims to generate a summary based on the multimodal context of conversations, clearly and intuitively describing the reasons that trigger a given emotion. To adapt to this task, we develop a new dataset (ECEM) based on the MELD dataset. ECEM combines video clips with detailed explanations of character emotions, helping to explore the causal factors behind emotional expression in multimodal conversations. A novel approach, FAME-Net, is further proposed, that harnesses the power of Large Language Models (LLMs) to analyze visual data and accurately interpret the emotions conveyed through facial expressions in videos. By exploiting the contagion effect of facial emotions, FAME-Net effectively captures the emotional causes of individuals engaged in conversations. Our experimental results on the newly constructed dataset show that FAME-Net outperforms several excellent baselines. Code and dataset are available at https://github.com/3222345200/FAME-Net.