LyS at SemEval-2024 Task 3: An Early Prototype for End-to-End Multimodal Emotion Linking as Graph-Based Parsing

TL;DR

This paper presents an early prototype system for multimodal emotion cause analysis in conversations, utilizing graph-based parsing and transformer encoders to identify causal emotion relations.

Contribution

It introduces a novel end-to-end multimodal system employing graph-based parsing and transformer models for emotion cause analysis in conversations.

Findings

Ranked 7th out of 15 in Subtask 1 using textual data.

Developed a neural transformer encoder for multimodal context.

Implemented a graph-based decoder for causal relation detection.

Abstract

This paper describes our participation in SemEval 2024 Task 3, which focused on Multimodal Emotion Cause Analysis in Conversations. We developed an early prototype for an end-to-end system that uses graph-based methods from dependency parsing to identify causal emotion relations in multi-party conversations. Our model comprises a neural transformer-based encoder for contextualizing multimodal conversation data and a graph-based decoder for generating the adjacency matrix scores of the causal graph. We ranked 7th out of 15 valid and official submissions for Subtask 1, using textual inputs only. We also discuss our participation in Subtask 2 during post-evaluation using multi-modal inputs.