Evaluation of data inconsistency for multi-modal sentiment analysis

TL;DR

This paper investigates the impact of semantic inconsistencies across modalities in multi-modal sentiment analysis, highlighting performance challenges for existing models and MLLMs, and introduces a conflicting test set for evaluation.

Contribution

It introduces a modality conflicting test set and evaluates the robustness of traditional and large language models in multi-modal sentiment analysis.

Findings

Traditional models perform poorly on conflicting data

MLLMs also face challenges with multi-modal emotion analysis

The study highlights the need for more robust multi-modal sentiment models

Abstract

Emotion semantic inconsistency is an ubiquitous challenge in multi-modal sentiment analysis (MSA). MSA involves analyzing sentiment expressed across various modalities like text, audio, and videos. Each modality may convey distinct aspects of sentiment, due to subtle and nuanced expression of human beings, leading to inconsistency, which may hinder the prediction of artificial agents. In this work, we introduce a modality conflicting test set and assess the performance of both traditional multi-modal sentiment analysis models and multi-modal large language models (MLLMs). Our findings reveal significant performance degradation across traditional models when confronted with semantically conflicting data and point out the drawbacks of MLLMs when handling multi-modal emotion analysis. Our research presents a new challenge and offer valuable insights for the future development of sentiment analysis systems.