EmoVerse: Exploring Multimodal Large Language Models for Sentiment and Emotion Understanding

TL;DR

EmoVerse is a multimodal large language model designed for comprehensive sentiment and emotion understanding, capable of analyzing emotional causes and outperforming existing methods across various affective computing tasks.

Contribution

The paper introduces EmoVerse, a novel MLLM that unifies sentiment and emotion tasks and employs multi-task training strategies for improved affective computing performance.

Findings

EmoVerse achieves state-of-the-art results on multiple affective computing tasks.

The Affective Multitask Dataset supports diverse emotion-related tasks.

EmoVerse effectively analyzes underlying emotional causes.

Abstract

Sentiment and emotion understanding are essential to applications such as human-computer interaction and depression detection. While Multimodal Large Language Models (MLLMs) demonstrate robust general capabilities, they face considerable challenges in the field of affective computing, particularly in detecting subtle facial expressions and handling complex emotion-related tasks, such as emotion reason inference and understanding emotions in long-context scenarios. Furthermore, there is a lack of a unified MLLM that can effectively handle both sentiment and emotion-related tasks. To address these challenges, we explore multi-task training strategies for MLLMs in affective computing and introduce Emotion Universe (EmoVerse), an MLLM designed to handle a broad spectrum of sentiment and emotion-related tasks. In addition, EmoVerse is capable of deeply analyzing the underlying causes of emotional states. We also introduce the Affective Multitask (AMT) Dataset, which supports multimodal sentiment analysis, multimodal emotion recognition, facial expression recognition, emotion reason inference, and emotion cause-pair extraction tasks. Extensive experiments demonstrate that EmoVerse outperforms existing methods, achieving state-of-the-art results in sentiment and emotion-related tasks. The code is available at https://github.com/liaolea/EmoVerse.